Building Nix packages and saving them in Glitch assets

The Gallery
1

Last thread: Install prebuilt packages without root, from nixpkgs

Here are some notes about my recent work on using the Nix package manager on Glitch.

Entirely-assets binary cache storage

My previous attempt at running a binary cache on Glitch was to store the actual .nar files in Glitch assets and everything else in a static site. I’ve since found that you can have slashes in asset names, making it possible to access as if there were a directory structure. That means we can set up a cache entirely in Glitch assets.

Having the cache entirely in Glitch assets is nice because (1) asset downloads don’t count toward your project’s hourly request limit and (2) it’s very easy to manage assets programmatically using Snail.

I’ve gone and done this. See,

If Glitch complains that we’re using the more expensive cdn.glitch.me domain, tell them it’s because cdn.glitch.global won’t serve anything larger than 20 MiB and some packages are larger than that.

Much normal-er binary cache infrastructure

I’ve since read up on what it would take to get nix copy --to (somewhere) to work, and it’s not that bad. You have to handle HTTP HEAD/GET to check if the destination already has any of the stuff you’re copying and PUT to do the actual uploading. You can even make it authenticated by telling Nix to use a .netrc file. We can do that–I did do that.

I set up a new project called tmpnix-upload which handles these methods and stores the packages.

We now have to maintain overlays for nixpkgs

nixpkgs nowadays distributes a newer glibc than what can naturally work on Glitch’s container system (see paranoid ramblings). I read up on Nix’s “overlay” system to add a patch to make glibc work again (thankfully someone else had already written the patch too).

So now we’re doubly needing to compile our own packages: (1) we’re using a different store path, and (2) we’re using a custom glibc, which is pretty far up on the entire repository’s dependency graph.

Note that there had already been packages that don’t successfully compile on Glitch, but they would at least still run okay. In this case, stuff wouldn’t even run properly.

This also makes it more useful that we moved the instantiation step to its own project, tmpnix-drv. The overlay thus has a dedicated place to live, and all other components make requests to tmpnix-drv and get properly patched derivations.

A more detailed diagram

Because things are slightly less janky, I’ve drawn up a more detailed diagram of how everything works together.

tmpnix-2022.drawio
tmpnix-2022.drawio1681×421 38.1 KB

Components of tmpnix. See below for walkthrough. Edit this in diagrams.net.

  1. A script I wrote called getdrv.sh makes a PUT request to the tmpnix-drv project, specifying the file to use (always nixpkgs for now) and the “attribute” within that file that has the “derivation” you want.
  2. An HTTP handler runs nix-instantiate on that file and attribute.
  3. nix-instantiate reads from the local copy of nixpkgs.
  4. nix-instantiate writes out the derivation and its “closure” of dependencies to the local store, in /tmp/nix/store.
  5. The handler then runs nix copy --to to get the derivations into a format that we can serve over HTTP.
  6. nix copy --to copies from the local store.
  7. nix copy --to signs the derivations with a store key.
  8. nix copy --to copies to a temporary area, /tmp/drvs
  9. getdrv.sh runs nix copy --from to download the derivations.
  10. nix copy --from copies from the tmpnix-drv project.
  11. An HTTP handler serves the derivations from the temporary area.
  12. nix copy --from verifies the derivations with a store key.
  13. nix copy --from copies to the local store.
  14. getdrv.sh outputs the requested derivation’s “store path,” and the builder passes that store path to nix-store -r.
  15. nix-store -r reads the derivation and its dependencies from the local store.
  16. nix-store -r gets already-built dependencies from a “substituter.”
  17. The substituter downloads packages stored in Glitch assets.
  18. The substituter verifies the packages with a store key.
  19. nix-store -r writes out the built packages to the local store.
  20. nix-store -r runs a post-build “hook” I wrote.
  21. The post-build hook runs nix copy --to to upload the built packages to the tmpnix-uploader project.
  22. nix copy --to copies from the local store.
  23. nix copy --to checks which packages tmpnix-uploader project already has.
  24. An HTTP handler redirects these read requests to the Glitch assets.
  25. nix copy --to signs the packages with a store key.
  26. nix copy --to copies to the tmpnix-uploader project.
  27. An HTTP handler writes the received packages to a temporary area, /tmp/upload-....
  28. The HTTP handler runs snail asset push to upload the packages to Glitch assets.
  29. snail asset push reads the packages from the temporary area.
  30. snail asset push uploads the packages to Glitch assets.

Onward to stable 22.05

Here are somethings I’m not looking forward to in the next nixpkgs stable to be released with NixOS 22.05 on 30 May 2022:

  • a newer release of the nix command line tool which tries harder to make you not use unstable commands like nix copy and nix show-derivation which I heavily rely on :skull:
  • even newer glibc, which I vaguely heard something about clone3 will further break :skull::skull:

… which is why I must write this up today and formally take a break from this project before that happens.

10 Likes
2

This is so cool :eyes:

6 Likes
3

22.05 is out as scheduled, glibc 2.34 is in, pthreads are broken as anticipated :scream:

2 Likes
4

LD_PRELOAD maybe?

5

would need a description of how

6

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.