Last thread: Install prebuilt packages without root, from nixpkgs
Here are some notes about my recent work on using the Nix package manager on Glitch.
Entirely-assets binary cache storage
My previous attempt at running a binary cache on Glitch was to store the actual .nar files in Glitch assets and everything else in a static site. I’ve since found that you can have slashes in asset names, making it possible to access as if there were a directory structure. That means we can set up a cache entirely in Glitch assets.
Having the cache entirely in Glitch assets is nice because (1) asset downloads don’t count toward your project’s hourly request limit and (2) it’s very easy to manage assets programmatically using Snail.
I’ve gone and done this. See,
https://cdn.glitch.me/35f4f809-f949-484d-af9c-269b3b7abc78/nar/18p4p9y06hf4i1yfffvvw4xsb1jrm097jlv4bfpr22w5yyqx5kck.nar.xz (note the
nar/18p4...slash in the asset name).
If Glitch complains that we’re using the more expensive
cdn.glitch.me domain, tell them it’s because
cdn.glitch.global won’t serve anything larger than 20 MiB and some packages are larger than that.
Much normal-er binary cache infrastructure
I’ve since read up on what it would take to get
nix copy --to (somewhere) to work, and it’s not that bad. You have to handle HTTP HEAD/GET to check if the destination already has any of the stuff you’re copying and PUT to do the actual uploading. You can even make it authenticated by telling Nix to use a .netrc file. We can do that–I did do that.
I set up a new project called tmpnix-upload which handles these methods and stores the packages.
We now have to maintain overlays for nixpkgs
nixpkgs nowadays distributes a newer glibc than what can naturally work on Glitch’s container system (see paranoid ramblings). I read up on Nix’s “overlay” system to add a patch to make glibc work again (thankfully someone else had already written the patch too).
So now we’re doubly needing to compile our own packages: (1) we’re using a different store path, and (2) we’re using a custom glibc, which is pretty far up on the entire repository’s dependency graph.
Note that there had already been packages that don’t successfully compile on Glitch, but they would at least still run okay. In this case, stuff wouldn’t even run properly.
This also makes it more useful that we moved the instantiation step to its own project, tmpnix-drv. The overlay thus has a dedicated place to live, and all other components make requests to tmpnix-drv and get properly patched derivations.
A more detailed diagram
Because things are slightly less janky, I’ve drawn up a more detailed diagram of how everything works together.
Components of tmpnix. See below for walkthrough. Edit this in diagrams.net.
- A script I wrote called
getdrv.shmakes a PUT request to the tmpnix-drv project, specifying the file to use (always nixpkgs for now) and the “attribute” within that file that has the “derivation” you want.
- An HTTP handler runs
nix-instantiateon that file and attribute.
nix-instantiatereads from the local copy of nixpkgs.
nix-instantiatewrites out the derivation and its “closure” of dependencies to the local store, in
- The handler then runs
nix copy --toto get the derivations into a format that we can serve over HTTP.
nix copy --tocopies from the local store.
nix copy --tosigns the derivations with a store key.
nix copy --tocopies to a temporary area,
nix copy --fromto download the derivations.
nix copy --fromcopies from the tmpnix-drv project.
- An HTTP handler serves the derivations from the temporary area.
nix copy --fromverifies the derivations with a store key.
nix copy --fromcopies to the local store.
getdrv.shoutputs the requested derivation’s “store path,” and the builder passes that store path to
nix-store -rreads the derivation and its dependencies from the local store.
nix-store -rgets already-built dependencies from a “substituter.”
- The substituter downloads packages stored in Glitch assets.
- The substituter verifies the packages with a store key.
nix-store -rwrites out the built packages to the local store.
nix-store -rruns a post-build “hook” I wrote.
- The post-build hook runs
nix copy --toto upload the built packages to the tmpnix-uploader project.
nix copy --tocopies from the local store.
nix copy --tochecks which packages tmpnix-uploader project already has.
- An HTTP handler redirects these read requests to the Glitch assets.
nix copy --tosigns the packages with a store key.
nix copy --tocopies to the tmpnix-uploader project.
- An HTTP handler writes the received packages to a temporary area,
- The HTTP handler runs
snail asset pushto upload the packages to Glitch assets.
snail asset pushreads the packages from the temporary area.
snail asset pushuploads the packages to Glitch assets.
Onward to stable 22.05
Here are somethings I’m not looking forward to in the next nixpkgs stable to be released with NixOS 22.05 on 30 May 2022:
- a newer release of the
nixcommand line tool which tries harder to make you not use unstable commands like
nix show-derivationwhich I heavily rely on
- even newer glibc, which I vaguely heard something about clone3 will further break
… which is why I must write this up today and formally take a break from this project before that happens.