Project URL: Glitch :･ﾟ✧
I’ve been trying out Nix, that package manager that installs everything into its own little directory, so that different versions of things don’t conflict with each other.
It’s shaping up to be a nice way to get some packages that are newer than the what we have from the 5-year-old distribution of Ubuntu we have in the project containers.
There’s still more to do, but now’s a good time to write up a post about what’s working already and how.*
Some merits and why Nix would be a good fit here
- It works without root. We can’t run as root on Glitch, so we need this.
- It installs things in its own area. We only have a few places we can write to disk, so we need this.
- The packages don’t depend on random stuff from the rest of the system. So they won’t mind if, for example, our gcc and libc are very old.
- It has pretty good tooling for letting you distribute prebuilt packages. Glitch is designed to reinstall dependencies pretty frequently, so it would be good not to have to compile everything.
The main problem so far: /nix/store
As I was saying, Nix installs stuff into its own area, out of the way of the rest of the system. That area is normally /nix/store. We can’t create that directory out in the root on Glitch. There are various ways you can set things up to work around this:
- Chroot somewhere. We’re not allowed to chroot.
- Use mount namespaces. We’re not allowed to set up mount namespaces.
- Run everything in a wrapper that uses ptrace to intercept file accesses and redirect them. We’re not allowed to use ptrace.
I concluded that it’s not in the cards for us to have /nix.
Where can we have it then?
/tmp/nix/store, IMO.
Not /app or /app/.data. It’s uncool to waste persistent storage on what’s essentially going to be dependencies.
/app/node_modules is kind of appealing, but it’s still limited to 1 GB. Here’s me right now in the middle of building gcc-10.3.0:
$ du -sh /tmp/nix
1008M /tmp/nix
Also, whereas maxing out /app would inconvenience you, maxing out /app/node_modules is a bigger no-no. According to other posts on the forum, there’s no friendly ENOSPC. Glitch will suspend your project 
.
These days software is more accepting of the Nix store being symlinked though, so maybe there won’t be much of a difference in practice. If you’re daring, you could have it in node_modules and make a symlink in /tmp/nix/store pointing there.
What does it cost to have it somewhere other than /nix/store?
I read up on why there’s so much emphasis on installing specifically in /nix/store. Nix installs each version of a piece of software is into its own dedicated directory, /nix/store/(a hash)-(name)-(version). Then they refer to their dependencies by these absolute paths. They say that, to make things work in a different location, we must recompile everything so that they can have these baked-into-the-package dependency paths point to the adjusted location.
Recompile everything.
We wouldn’t be able to use Nix’s prebuilt packages, which were built for /nix/store. We’d have to compile packages from source and supply our own infrastructure for distributing prebuilt packages. That’s kind of a big task.
But you know what, here are some other nice things about this situation that make the task less daunting:
- The packages are defined parametrically, and we won’t be wading through build scripts to change this installation directory.
- Everything is so tooled up to handle the might-as-well-be-unpredictable
/nix/store/(a hash)-(name)-(version)paths that it all works great when we redo it with a different prefix. - Glitch gives us an asset CDN with plenty of space to store the compiled packages.
- Glitch gives us project containers that we can use to compile things.
How do we bootstrap?
There’s a bit of a chicken-and-egg problem before we get going. The piece of software that manages how to build all these packages and install them into that special place is a package manager called nix.
The usual way to install nix (NixOS - Getting Nix / NixOS) is to set it up as a Nix package, i.e., in /nix/store.
So we need another way to set up nix, a way that doesn’t involve first having nix. Here’s the plan:
- Do whatever it takes to get a copy of
nix(a) running barely enough for the next steps. - Have (a) operate on store
/tmp/nix/store. - Use (a) to install
nix(b) properly. - Do everything after that with (b), which should fully work.
I tried a few things for step 1’s “whatever it takes.”
Getting a barely working copy of nix
Install it from Ubuntu.
Ubuntu packages nix, so can we download it and run it? They started shipping it in Ubuntu 20.10. We don’t have that on Glitch of course, because our humble project containers are on 16.04. And its dependencies aren’t satisfied by what we have in our project container.
So that didn’t work.
Build it from source.
nix is open source, so can we compile it ourselves? Not on Glitch, we can’t. The build tools (g++ et al.) are too far out of date, according to their configure script.
So that didn’t work.
Build it on a different computer with a recent compiler.
I tried it, but then it wouldn’t run because it compiled such that it would depend on the various libraries on that other computer, and it wouldn’t run on Glitch.
So that didn’t work.
Download a statically linked copy.
The first-party package index for Nix, called nixpkgs, has a static build of nix. Couldn’t we run that? I feel like this should have worked, but something was out of whack with the static version’s string library, and it can’t download the package index. This was reported (`nix-env --upgrade` upgrades 'nix-2.3.10' to 'nix-2.3.10-x86_64-unknown-linux-musl' · Issue #118481 · NixOS/nixpkgs · GitHub), but they closed it after solving an unrelated problem of non-static nix being upgraded to nixStatic automatically. I guess we’ll never get to the bottom of this.
So that didn’t work.
Patch the official nix package to fix up the dependency paths.
My current approach is to patch the installer’s copy to work in a different path. I use the Nix project’s own patchelf tool to rewrite all the binaries’ and libraries’ rpath and, for the binaries, the interpreter. Then there’s a set of paths for nix to execute other programs, which, thank goodness, are configurable in some environment variables and configuration files.
This one worked.
Additional notes on the journey of getting nix (a) to install nix (b)
binutils-2.35.1
63m56.335s
binutils took over an hour to build. That, by the way, is the reference ‘SBU’ in LFS. Slow. This is the effect of Glitch’s pretty darn low CPU quota.
gcc-10.3.0
GCC (albeit version 11; Nixpkgs seems to be on 10) in LFS sits at 28 SBU. Holy.
This one took 660 minutes. Glitch at one point said that project containers restart at least every 12 hours. So that’s 92% of a project container’s nominal lifetime. If you want to try this on Glitch, make sure you have really low uptime before you start.
That was a heck of a long day, where if I didn’t keep wiggling the mouse on the editor, then I would suddenly lose 11 hours of progress.
No, you know what, it didn’t just work out on the first try. My first attempt didn’t have the right parallelism set, and it went over the RAM limit, and I lost my progress. My second attempt got about 8 hours in, and I had to go out, and I lost my progress. My third attempt succeeded, but it was tense.
coreutils-8.32
One of the tests (chown would make sense, but I don’t remember exactly) fails if building in an environment where you’re not listed in the passwd and have no username. This was during a time where I tried compiling some stuff on a faster computer but which didn’t have an /etc/passwd set up right. I switched back to compiling on Glitch after encountering this. Good ol’ app user.
aws-c-common-0.5.11
They set a CPU affinity for their background logging thread, which gives EINVAL on Glitch. The slapdash test harness overlooks that the logging init fails, resulting in cryptic errors such as this
Signal received: 0, errno: 0
################################################################################
Stack trace:
################################################################################
/tmp/nix-build-aws-c-common-0.5.11.drv-0/source/build/libaws-c-common.so.1(aws_backtrace_print+0x62) [0x7fc905832c42]
./aws-c-common-tests() [0x4643b3]
/tmp/nix/store/zg9asy4mbbjz057d77pv20npb59dx5fg-glibc-2.32-46/lib/libpthread.so.0(+0x13700) [0x7fc9056b2700]
/tmp/nix-build-aws-c-common-0.5.11.drv-0/source/build/libaws-c-common.so.1(aws_logger_clean_up+0x3) [0x7fc90582f413]
./aws-c-common-tests() [0x467fa1]
./aws-c-common-tests() [0x409684]
/tmp/nix/store/zg9asy4mbbjz057d77pv20npb59dx5fg-glibc-2.32-46/lib/libc.so.6(__libc_start_main+0xed) [0x7fc905503ded]
./aws-c-common-tests(_start+0x2a) [0x4098aa]
when the clean up routine follows an uninitialized pointer hoping for a vtable.
I had to build this package on a different computer.
aws-sdk-cpp-1.8.121
Wow this one takes a lot of memory to compile.
1.503g on ub_S3.cpp.o
Our copy of nix (b)
Compiling nix and all its build tools and dependencies is the largest project I’ve undertaken on Glitch. Here’s the result.
https://cdn.glitch.com/f7a45a64-644d-458a-9b4d-f84cc2a5cc97%2Fnix-g.tar.gz?v=1624131910455
A copy of nix that can run from /tmp/nix/store.
Note that although it’s installed in a different location, we haven’t made any changes to the package manager itself. This is stock nix, meaning it by default expects to install things to /nix/store. We use the following environment variables to have it install into our adjusted location:
export NIX_STORE_DIR=/tmp/nix/store
export NIX_LOG_DIR=/tmp/nix/var/log/nix
export NIX_STATE_DIR=/tmp/nix/var/nix
export USER=app
Oh and for whatever little reasons, Glitch doesn’t run login, so there’s no USER environment variable, which confuses some parts of Nix. Set that too 
.
Our own binary cache infrastructure
The process of building everything up until nix took many days, and for disk space reasons I had to do it in /tmp. So I had to hack together some stuff to save my progress. Nix has a lot of tools that are useful for this.
An early script I had would use nix-store --export and nix-store --import with an ever growing list of packages that I had already compiled.
Later I wrote a fancier “build hook” that would use nix copy to make an on-disk cache and then upload it to the asset CDN. These prebuilt packages are up at https://tmpnix-cache.glitch.me/ and they’re ready to be used.
* because the project got suspended 
. I’ve sent a request to ask why. This post serves (1) to describe what was in the project, in case that helps the Glitch team figure out why it was suspended, (2) to describe how things worked so people can look at it and help me make changes to fix what caused a violation, and (3) as a sign of good faith, show the various measures I took to ensure that all this would fit in within the various terms of service.
