Actually building Nix packages on Glitch

The Gallery
2

Other build problems from 22.05 and before

A quick addendum on the last post: weren’t there a few other packages that couldn’t be built on Glitch back when I was building the 22.05 release Things encountered while building nixpkgs 22.05? Yes. But for some reason they build now.

  • Git wouldn’t build due to some exotic malloc-instrumenting LD_PRELOAD that it wanted to do in its test suite (did I remember to say? the tests are part of the build process), which somehow didn’t work right. I looked through the recent changes in Git’s source code and Nixpkgs, but I didn’t find anything that would have fixed the problem. Yet it built fine this time. This is going to be one of those things that I never figure out. Ugh!
  • pytest-asyncio had a test that (almost forgot to say, the tests are part of the build process) would try to listen on an IPv6 address. Well somehow that wasn’t a problem this time. Maybe something fixed something? Eh.

Cloneable builders

As you may have noticed, the link in the previous post goes to a project called “tmpnix-cb1.” That “1” is there because there’s now more than one project doing the building.

Deploying files with secrets

Previously I had split up the build process into a big multi-project system of derivation instantiator, builder, and uploader. That made some things complicated, because the various parts needed all sorts of credentials to authenticate with each other: Nix signing keys, netrc files, and Snail persistent token files. These are all in the .data directory so that remixes (they’re public projects, after all) don’t get copies.

As a result, it used to take many steps to set up another project to use as a builder, creating multiple various files in .data in the project terminal.

I’ve simplified the process of setting up the various secret-containing files by shipping an encrypted tarball containing all of those files I need in .data. Now, there’s still some manual setup, but it’s all in one step with much fewer switching around tabs. All I do is decrypt and extract the big file with a password that’s saved in the main project’s .env file.

Saving a password in .env

I needed a place to put it so that I could easily copy it from there. .env seems nice, because it shows up in the editor. A downside though is that it puts it into the environment of all your processes, which seems unnecessarily risky. The processes don’t need to know the password.

I’m working around this “things in .env actually go into the environment” issue by doing this in .env:

SECRETS_PASSPHRASE=...
# don't actually tell every process lol
SECRETS_PASSPHRASE=1

Thus processes only see SECRETS_PASSPHRASE=1. Unless they specifically go looking in .env, but I can’t do anything about that, haha.

Deploying dotfiles

There are some dotfiles that are usually ignored, but which I want to replicate in these cloneable builders. So far, that’s these:

  • .config/nix: Nix config
  • .config/nixpkgs: my Nixpkgs overlays
  • .emacs: my precious editor settings lol. Mostly to tell Emacs to use a dark theme. It boggles the mind that it defaults to using a light theme when running in a terminal. And “using a light theme” here doesn’t even mean drawing dark text on a light background. It simply uses dark colors on what is still a black terminal background :new_moon_with_face:
  • .gitconfig: it’s got my username and command aliases :sunglasses:. Note that this is my “global” Git config as a dotfile in the app user’s home directory, distinct from .git/config.
  • .gitignore: this is the file that’s going to be programming the inclusion of these dotfiles, so
  • .profile: needed for setting up a Nix environment. Basically it puts the stuff installed into your “profile” on your PATH. Possibly a few other things, I should learn what.

.emacs, .gitconfig, and .gitignore are tracked by default. so there’s no issue including them.

.profile you can put a !/.profile in your .gitignore easily enough.

There’s a trick to un-ignoring .config/nix and .config/nixpkgs. /etc/gitignore-global includes .config as the whole directory. The way Git works, that makes it not explore the directory at all, so even if you put !/.config/nix in your .gitignore, it still wouldn’t pick it up.

You have to un-ignore the directory (!/.config), ignore everything else inside it (./config/*), and then unignore the things you want to track (!/.config/nix, !/.config/nixpkgs).

Weird thing I encountered while setting up dotfiles’ ignored-ness: Glitch by default includes .gnupg in remixes. So if you’ve generated a GPG key on Glitch (in a public project), uh-oh.

You can kind of make these files visible in the editor too. Un-ignoring them doesn’t cause them to be included in the editor, but what I’ve done is to make the dotfiles symlinks to other file names that would normally show up in the editor. For example:

$ ls -al .config
total 4
...
lrwxrwxrwx  1 app app   28 Mar  6 05:55 nix -> ../tmpnix/visible.config/nix
...

Screen Shot 2023-03-11 at 12.19.04

Coordinating which project builds what

Nix has its own system for running builds remotely. It’s done over SSH. Nix has a little protocol to send over the build dependencies, tell the remote builder to build, and send back the build outputs. But I don’t have all the pieces in place to use it. I have some experiments connecting to Glitch projects over SSH Connecting via SSH, editing a project in VS Code. For one thing, I haven’t gotten it all set up here. Besides that, there’s this lurking evil of sending sizeable build dependencies and outputs over the Glitch terminal, which uh… does not have backpressure Experience report: building an administrative connection to Glitch - #5 by wh0. Oh and there’s the obvious problem of Glitch project containers going to sleep when the editor isn’t open. So I’m not doing this coordination in the Nix remote build way.

Currently I’m doing this:

  1. Have the first builder try to build everything (nix-store -rk ...).
  2. When it gets to a big package that I know takes a long time to build, I interrupt that build (it suffices to kill the running make or whatever).
  3. The -k in the nix-store -r command causes the first builder to carry on building other packages that it can build.
  4. I manually (lol) open up a second builder and run a command to build specifically that big package. The build dependencies for this package are ready, because the first builder tried to build it. Thus the second builder only has to build this one package.
  5. The second builder finishes and uploads the outputs to the shared cache.
  6. Eventually the first builder finishes everything it can do and reports that building the big package failed.
  7. I rerun the command to build everything on the first builder, and it discovers that the big package is available from the cache. So it continues building other things that depend on that big package.

And there are enough known “big” packages to get good time savings this way:

  • gcc (~2.5h)
  • ghc (~3h)
  • llvm (~6.5h)
  • clang (~4h)
  • rustc (~2.5h)
  • openjdk-headless (~1h)
  • nodejs (~6.5h)

But it’s kind of a pain to be switching through multiple editor tabs to wiggle the mouse around in each of them.

dramatization of me keeping more than one Glitch editor open

In the next post, I’ll talk about re-bootstrapping the whole build.