Good things from Nix that NAR Flinger doesn’t do
- Nix maintains a database of what’s installed, including what depends on what. And that’s great if you later need to uninstall things. For our use case though, we’re installing into /tmp, which will be cleared each time Glitch recreates your project container. So things won’t be installed for more than 12 hours anyway. Although unfortunately, this makes NAR Flinger not actually compatible with Nix.
- Nix builds “profiles,” as mentioned above, which aggregate a bunch of symlinks into a directory structure. Importantly, it’s not just for the binaries. You also get symlinks to, for example, manpages. NAR Flinger so far only creates the symlinks for binaries. I’m interested in expanding this to other kinds of files.
- Nix can evaluate the Nix programming language, which is involved in a very normal way Nix users specify which packages they want to install etc. Normally they specify an “attribute” name, in a great big dictionary of those names to package definitions. That great big dictionary is defined by a program in a functional programming language—the Nix language. And being able to refer to a package by that attribute name is what lets normal users say they want “nodejs” instead of something as detailed as “r9gqywii82drj1m893kdsdkidc80ir9z-nodejs-18.14.1” as in our sample app. It’s important that NAR Flinger doesn’t do this though, because that great big dictionary of package definitions includes some things that Glitch will suspend your project for. But I am interested in publishing a list of these package names that you can use in NAR Flinger.
- Nix can build packages from source. That’s useful for when you want to customize something about a package. For NAR Flinger, I felt it would slow down the project installation too much if we were to compile things from source. If you must use a customized package build them using the real Nix and upload them to a binary cache for NAR Flinger to install from. Actually the current suite of software available from my binary cache contains some patches to make modern glibc work on Glitch (read more).
- Nix performs integrity/authenticity checks on the packages you download from a binary cache. That’s why you set up a public key when using a custom binary cache containing packages built by someone other than the NixOS project. NAR Flinger doesn’t do these, so you’ll have to rely on the integrity of the binary cache host, which in this case is Glitch’s asset CDN and GitHub for packages that don’t fit on Glitch’s asset CDN (read more).
- Nix makes the “store” read-only. Which is fine, because in Nix you don’t edit or overwrite things. When you edit a package by changing the source or changing the dependencies or adding patches, Nix computes the updated cryptographic hash of all sources and instructions, resulting in a new location to store the results of the build. NAR Flinger doesn’t mark everything as read-only. There’s no special reason not to. Although I’ll note that it made resetting things during development simpler.
NAR Flinger internals
NAR Flinger by default uses /tmp/nix/store and my binary cache, which are suitable for use on Glitch. But you can specify narflinger.store_prefix and narflinger.store_prefix and narflinger.base to override these, respectively.
At the core of it is routines to crawl a package’s dependency tree in a binary cache and to parse the contents of a NAR file. These I ported from the Node.js version that was a precursor to the current Python implementation (read more). And that Node.js version I adapted from a script for web browsers (read more).
If the search for dependencies encounters a package that’s already in the store, it skips installing it and searching for dependencies of that package.
NAR Flinger collects the list of packages to install using a depth-first search, following each package’s dependencies. It visits these packages in post-order, so that it installs a package’s dependencies before installing the package. This helps keep the contents of the Nix store in a state that Nix calls “valid” (except for that part about the database, NAR Flinger doesn’t maintain a database), meaning that a package in the store has its dependencies in the store too.
When parsing through a NAR file, NAR Flinger writes its contents out to a temporary directory. When it finishes unpacking it, it renames the directory to its store path. That prevents it from creating any partial packages in the store.
After it unpacks a package that’s requested in the package.json, it links the the files in the package’s bin directory into your ~/.local/bin directory. Note that the files in the bin directories of dependencies don’t get linked this way, so if you want something make sure it’s explicitly listed in your package.json. If multiple packages listed in your package.json provide a given name in the bin directory, packages later in the list will win.
Weird thing when using this to install Node.js
When your project first starts up after Glitch sets up a new project container for it, Glitch uses the old built-in version of npm and Node.js to start the npm install command. So it’ll complain about the package-lock.json being in a future version that it doesn’t know about. Then the install script runs to install the new version of Node.js and npm.
If the install step runs again, for example if you edit your package.json, then the new version of npm and Node.js get used, which then complains about the old version of package-lock.json.
How might we work around this?
I’m going to try making some more projects to demo other modern runtimes. I’m have the packages listed in the NixOS “small” set built (read more), so there should be a good few to try out.