Install prebuilt packages without root, from nixpkgs

Project URL: Glitch :・゚✧

I’ve been trying out Nix, that package manager that installs everything into its own little directory, so that different versions of things don’t conflict with each other.

https://nixos.org/

It’s shaping up to be a nice way to get some packages that are newer than the what we have from the 5-year-old distribution of Ubuntu we have in the project containers.

There’s still more to do, but now’s a good time to write up a post about what’s working already and how.*

Some merits and why Nix would be a good fit here

  1. It works without root. We can’t run as root on Glitch, so we need this.
  2. It installs things in its own area. We only have a few places we can write to disk, so we need this.
  3. The packages don’t depend on random stuff from the rest of the system. So they won’t mind if, for example, our gcc and libc are very old.
  4. It has pretty good tooling for letting you distribute prebuilt packages. Glitch is designed to reinstall dependencies pretty frequently, so it would be good not to have to compile everything.

The main problem so far: /nix/store

As I was saying, Nix installs stuff into its own area, out of the way of the rest of the system. That area is normally /nix/store. We can’t create that directory out in the root on Glitch. There are various ways you can set things up to work around this:

  • Chroot somewhere. We’re not allowed to chroot.
  • Use mount namespaces. We’re not allowed to set up mount namespaces.
  • Run everything in a wrapper that uses ptrace to intercept file accesses and redirect them. We’re not allowed to use ptrace.

I concluded that it’s not in the cards for us to have /nix.

Where can we have it then?

/tmp/nix/store, IMO.

Not /app or /app/.data. It’s uncool to waste persistent storage on what’s essentially going to be dependencies.

/app/node_modules is kind of appealing, but it’s still limited to 1 GB. Here’s me right now in the middle of building gcc-10.3.0:

$ du -sh /tmp/nix
1008M   /tmp/nix

Also, whereas maxing out /app would inconvenience you, maxing out /app/node_modules is a bigger no-no. According to other posts on the forum, there’s no friendly ENOSPC. Glitch will suspend your project :scream:.

These days software is more accepting of the Nix store being symlinked though, so maybe there won’t be much of a difference in practice. If you’re daring, you could have it in node_modules and make a symlink in /tmp/nix/store pointing there.

What does it cost to have it somewhere other than /nix/store?

I read up on why there’s so much emphasis on installing specifically in /nix/store. Nix installs each version of a piece of software is into its own dedicated directory, /nix/store/(a hash)-(name)-(version). Then they refer to their dependencies by these absolute paths. They say that, to make things work in a different location, we must recompile everything so that they can have these baked-into-the-package dependency paths point to the adjusted location.

:grimacing: Recompile everything.

We wouldn’t be able to use Nix’s prebuilt packages, which were built for /nix/store. We’d have to compile packages from source and supply our own infrastructure for distributing prebuilt packages. That’s kind of a big task.

But you know what, here are some other nice things about this situation that make the task less daunting:

  1. The packages are defined parametrically, and we won’t be wading through build scripts to change this installation directory.
  2. Everything is so tooled up to handle the might-as-well-be-unpredictable /nix/store/(a hash)-(name)-(version) paths that it all works great when we redo it with a different prefix.
  3. Glitch gives us an asset CDN with plenty of space to store the compiled packages.
  4. Glitch gives us project containers that we can use to compile things.

How do we bootstrap?

There’s a bit of a chicken-and-egg problem before we get going. The piece of software that manages how to build all these packages and install them into that special place is a package manager called nix.

The usual way to install nix (NixOS - Getting Nix / NixOS) is to set it up as a Nix package, i.e., in /nix/store.

So we need another way to set up nix, a way that doesn’t involve first having nix. Here’s the plan:

  1. Do whatever it takes to get a copy of nix (a) running barely enough for the next steps.
  2. Have (a) operate on store /tmp/nix/store.
  3. Use (a) to install nix (b) properly.
  4. Do everything after that with (b), which should fully work.

I tried a few things for step 1’s “whatever it takes.”

Getting a barely working copy of nix

Install it from Ubuntu.
Ubuntu packages nix, so can we download it and run it? They started shipping it in Ubuntu 20.10. We don’t have that on Glitch of course, because our humble project containers are on 16.04. And its dependencies aren’t satisfied by what we have in our project container.

So that didn’t work.

Build it from source.
nix is open source, so can we compile it ourselves? Not on Glitch, we can’t. The build tools (g++ et al.) are too far out of date, according to their configure script.

So that didn’t work.

Build it on a different computer with a recent compiler.
I tried it, but then it wouldn’t run because it compiled such that it would depend on the various libraries on that other computer, and it wouldn’t run on Glitch.

So that didn’t work.

Download a statically linked copy.
The first-party package index for Nix, called nixpkgs, has a static build of nix. Couldn’t we run that? I feel like this should have worked, but something was out of whack with the static version’s string library, and it can’t download the package index. This was reported (`nix-env --upgrade` upgrades 'nix-2.3.10' to 'nix-2.3.10-x86_64-unknown-linux-musl' · Issue #118481 · NixOS/nixpkgs · GitHub), but they closed it after solving an unrelated problem of non-static nix being upgraded to nixStatic automatically. I guess we’ll never get to the bottom of this.

So that didn’t work.

Patch the official nix package to fix up the dependency paths.
My current approach is to patch the installer’s copy to work in a different path. I use the Nix project’s own patchelf tool to rewrite all the binaries’ and libraries’ rpath and, for the binaries, the interpreter. Then there’s a set of paths for nix to execute other programs, which, thank goodness, are configurable in some environment variables and configuration files.

This one worked.

Additional notes on the journey of getting nix (a) to install nix (b)

binutils-2.35.1

63m56.335s

binutils took over an hour to build. That, by the way, is the reference ‘SBU’ in LFS. Slow. This is the effect of Glitch’s pretty darn low CPU quota.

gcc-10.3.0

GCC (albeit version 11; Nixpkgs seems to be on 10) in LFS sits at 28 SBU. Holy.

This one took 660 minutes. Glitch at one point said that project containers restart at least every 12 hours. So that’s 92% of a project container’s nominal lifetime. If you want to try this on Glitch, make sure you have really low uptime before you start.

That was a heck of a long day, where if I didn’t keep wiggling the mouse on the editor, then I would suddenly lose 11 hours of progress.

:weary: No, you know what, it didn’t just work out on the first try. My first attempt didn’t have the right parallelism set, and it went over the RAM limit, and I lost my progress. My second attempt got about 8 hours in, and I had to go out, and I lost my progress. My third attempt succeeded, but it was tense.

coreutils-8.32

One of the tests (chown would make sense, but I don’t remember exactly) fails if building in an environment where you’re not listed in the passwd and have no username. This was during a time where I tried compiling some stuff on a faster computer but which didn’t have an /etc/passwd set up right. I switched back to compiling on Glitch after encountering this. Good ol’ app user.

aws-c-common-0.5.11

They set a CPU affinity for their background logging thread, which gives EINVAL on Glitch. The slapdash test harness overlooks that the logging init fails, resulting in cryptic errors such as this

Signal received: 0, errno: 0
################################################################################
Stack trace:
################################################################################
/tmp/nix-build-aws-c-common-0.5.11.drv-0/source/build/libaws-c-common.so.1(aws_backtrace_print+0x62) [0x7fc905832c42]
./aws-c-common-tests() [0x4643b3]
/tmp/nix/store/zg9asy4mbbjz057d77pv20npb59dx5fg-glibc-2.32-46/lib/libpthread.so.0(+0x13700) [0x7fc9056b2700]
/tmp/nix-build-aws-c-common-0.5.11.drv-0/source/build/libaws-c-common.so.1(aws_logger_clean_up+0x3) [0x7fc90582f413]
./aws-c-common-tests() [0x467fa1]
./aws-c-common-tests() [0x409684]
/tmp/nix/store/zg9asy4mbbjz057d77pv20npb59dx5fg-glibc-2.32-46/lib/libc.so.6(__libc_start_main+0xed) [0x7fc905503ded]
./aws-c-common-tests(_start+0x2a) [0x4098aa]

when the clean up routine follows an uninitialized pointer hoping for a vtable.

I had to build this package on a different computer.

aws-sdk-cpp-1.8.121

Wow this one takes a lot of memory to compile.

1.503g on ub_S3.cpp.o

Our copy of nix (b)

Compiling nix and all its build tools and dependencies is the largest project I’ve undertaken on Glitch. Here’s the result.

https://cdn.glitch.com/f7a45a64-644d-458a-9b4d-f84cc2a5cc97%2Fnix-g.tar.gz?v=1624131910455

A copy of nix that can run from /tmp/nix/store.

Note that although it’s installed in a different location, we haven’t made any changes to the package manager itself. This is stock nix, meaning it by default expects to install things to /nix/store. We use the following environment variables to have it install into our adjusted location:

export NIX_STORE_DIR=/tmp/nix/store
export NIX_LOG_DIR=/tmp/nix/var/log/nix
export NIX_STATE_DIR=/tmp/nix/var/nix

export USER=app

Oh and for whatever little reasons, Glitch doesn’t run login, so there’s no USER environment variable, which confuses some parts of Nix. Set that too :woman_shrugging:.

Our own binary cache infrastructure

The process of building everything up until nix took many days, and for disk space reasons I had to do it in /tmp. So I had to hack together some stuff to save my progress. Nix has a lot of tools that are useful for this.

An early script I had would use nix-store --export and nix-store --import with an ever growing list of packages that I had already compiled.

Later I wrote a fancier “build hook” that would use nix copy to make an on-disk cache and then upload it to the asset CDN. These prebuilt packages are up at https://tmpnix-cache.glitch.me/ and they’re ready to be used.


* because the project got suspended :cry:. I’ve sent a request to ask why. This post serves (1) to describe what was in the project, in case that helps the Glitch team figure out why it was suspended, (2) to describe how things worked so people can look at it and help me make changes to fix what caused a violation, and (3) as a sign of good faith, show the various measures I took to ensure that all this would fit in within the various terms of service.

7 Likes

Now this is an example of how to conduct research and how to document that research. Great job!

3 Likes

Refactoring “instantiate” and “build” pipeline

The support staff has helped me figure out why the previous project where I compiled packages is suspended. When we download the package index, that index contains the names of programs which can mine cryptocurrency. The presence of those names on disk in combination with other circumstances created by permitted uses causes an abuse detection heuristic to suspend the project.

I assume Glitch won’t publish the exact abuse detection heuristics. But even without exact knowledge of the rules, we can make it a lot more obvious that we’re not doing crypto mining.

The design of Nix turns out to be pretty handy in this case. In building a package from the package index, there are two steps in the process. First is a relatively quick step called “instantiation,” which takes a high level specification of the package, figures out the dependency graph, and produces a set of files called “derivations” containing detailed instructions on how to build the package and its dependencies. Second, the actual building will run those steps, which involves downloading source archives and running compilers, and which takes a long time for the larger packages. Nix has built in functionality for transmitting derivations over the network.

I’ve split up the build system into two projects—one for instantiation and one for building:

tmpnix-diagram

This leads to two nice properties:

  1. The project that instantiates will download the package index and thus contain references to software that can mine cryptocurrency, but it uses very little resources and doesn’t stay online for long.
  2. The project that builds will use its entire share of CPU for as long as it takes to build, but because we only copy over the relevant derivations, we can avoid having any references to cryptocurrency-specific software in the project.

Although if you wanted to build software that can mine cryptocurrency, I think this approach wouldn’t work. But there are cool things to do other than that.

Switching to Nixpkgs stable

The normal Nix installer by default sets up the Nixpkgs package index at a so-called “unstable” channel, which contains the latest software, updated for any kind of changes. Another “stable” channel gets “conservative updates for fixing bugs and security vulnerabilities” (Nix channels - NixOS Wiki).

I’ve liked rolling releases in other distributions, but the way Nix builds packages results in a lot of extra work for rolling releases. Nix packages are like nodes in a Merkle DAG of runtime- and build-dependencies. So if a there’s a minor version bump to a library referenced by another library used in a compiler used to build a dependency of your package, you’re gonna end up with a new copy of your package. And there’s no “just dynamically link to the new library” in Nix. As things are now, you’ll have to compile everything.

It must be a lot of work when there’s a change to glibc, which I believe must find itself within almost all package transitive-build-dependencies (I think the Nix term for that would be ‘derivation closure’ or something). Within the 6-month lifetime of the previous stable release, glibc had 5 releases. During that time, it had 22 releases in the unstable channel.

My appetite for recompiling everything day to day is toward the low end, so I’ll be using the stable channel going forward. Having new packages is one of the goals of this work, but I think we’re still getting that, relatively speaking. The current stable release is a release called 21.05, while the project containers are on a release called 16.04, both presumably using year.month style versioning.

Demo

random ASCII art bonsai tree powered by cbonsai
View source

What a neat program this is:

It’s actually packaged in Ubuntu (Ubuntu – Details of package cbonsai in impish), but we can’t install that in our project containers because we have a much older Ubuntu release, and we don’t have root access, which apt needs to do its package management.

Also, remember that thing about Ruby having problems with Glitch’s reverse proxy, where it turned out to be caused by an outdated version? That’s another place where it sure would be nice to have an updated package.

So I’ve put together an example app where I install cbonsai and Ruby from Nix. If you wanted a webpage that shows you a random ASCII art bonsai tree, it’s taken years to get to this point, but you can now have that on Glitch.

3 Likes