last thread: NAR Flinger, a package installer in a single script
Here it is! https://tmpnix-gsdrv.glitch.me/built.html
Screenshot:
Alright, the installer exists. The packages are built. Finally, here’s the list of built packages. For example, to install git, find that in the list:
/tmp/nix/store/yc00szqimcw43zpqyphdvcz0q6zch3a1-git-2.40.1.tar.xz.drv derivation preferLocalBuild
/tmp/nix/store/i9d2fy4q00a8pszrv9x5f82wm06s8g97-git-2.40.1.tar.xz output
/tmp/nix/store/g1qs0xfpqkfcdbmqm6hv5ff0zy8jz85j-git-2.40.1.drv derivation top
/tmp/nix/store/w8b32f2yvi4skkblpbmlmyf2q0b22w8k-git-2.40.1-debug output
/tmp/nix/store/dvd3svfvarbx1yyh3p80a56vsk7asq5h-git-2.40.1-doc output
/tmp/nix/store/zz4vy3p84ilw9xrw922k6c6bya7z3p3q-git-2.40.1 output
The one you want is the “zz4vy3p84ilw9xrw922k6c6bya7z3p3q-git-2.40.1” one. The first one, which ends in .tar.something.drv, is the instructions for how to download the source code for git. The second one, which ends in .tar.something, is the source code. The third one, which ends in .drv, is the instructions for how to build git. The fourth and fifth ones, which end in -something, are additional packages that come from the build process: the debug symbols and the documentation for git. The program itself is in that last one.
So take that “zz4vy3p84ilw9xrw922k6c6bya7z3p3q-git-2.40.1” (i.e. without the /tmp/nix/store/ path leading up to it) and paste it into the narflinger.basenames section of your package.json (read about NAR Flinger).
Details about what’s in the list
Markings
All derivations and source files needed for the packages I’ve compiled are listed. These files are marked as “derivation” and “source,” respectively. Derivations are the build instructions for a package. Source files here mean files that aren’t separately downloaded from, for example, a package’s website. They’re the files needed in the build process that are just provided along with the build instructions. Lots of Nix-specific patches are included in this way. The derivations’ recursive dependencies are included too.
The Nixpkgs project marks some derivations as “preferLocalBuild” where they think it’s more overhead to upload/download them than to build them whenever they’re needed, such as source code archives (“building” those downloads them anyway) and certain configuration files. Derivations marked as preferLocalBuild are marked as that thusly marked in the list.
The derivations at the roots of the dependency graph were programs that I explicitly chose to be compiled. In the list, they’re marked as “top” (apologies for the confusion for people who aren’t used to laying out their graphs from the top down).
After each derivation, the derivation’s outputs are listed, marked as “output.” The outputs are the built programs etc. that you’ll want to use with NAR Flinger. Derivations have a “default” output and can have other outputs too, which will appear with suffixes like “-dev” and “-man.”
Output availability
Although all derivation outputs are listed, some derivation outputs I haven’t uploaded. That’s usually the case for outputs of derivations that are preferLocalBuild. However, some exceptions where they are still uploaded.
First, if a preferLocalBuild derivation is one of the roots (i.e. marked as “top”), it’s still uploaded. For example, the package for Clang is called clang-wrapper (version 16 if you’re going to check; there are other version of Clang that appear in the dependency tree and other ones aren’t marked as “top”), and it’s indeed a wrapper around another package that has the actual Clang binaries. Nixpkgs supposedly finds that it’s more practical to build it than to upload it. But for us using NAR Flinger, we won’t be keeping the build tools around, so it’s much simpler to have it uploaded.
Second, if a preferLocalBuild derivation’s output is a dependency of another package that’s uploaded, then it’s also uploaded. That’s because a Nix cache must have all recursive dependencies of the packages it has in order to be valid.
The links
Oh and each entry is also a link to view that file/package in a viewer that I wrote earlier! Read more about that if you want 
Other section with no segue
I’m also writing a tool for automatically assigning compilation jobs to projects that run the build (read more about builder projects).
Other than the few big packages that take a long time to compile, there are tons of little packages where up until now it’s just been easiest to have a single project compile them all, so that I don’t have to go through the trouble of splitting them up across multiple builders. The compile time for all of these little packages add up too, so I’m pretty sure it’s faster to split them up too. (Hand-wave-y anecdotal claim: 4 days down to 2 days for probably the whole package list.)
(There’s no UI, it’s all console.log. That’s how strong I am at frontend development.) (Tech recruiters who find this post, you better not bring this up.)
I ended up writing a simple program—you know what, I’m going to call it an AI—to manage the build. It’s built on simple rules:
- If a builder has a job, connect to it to wake it up
- If a builder is done with its build job, remember that there might be more jobs that are ready to start (i.e. packages that depend on the package that we just finished building)
- If a builder meets certain conditions to start its build job, run some commands to start it
- If a builder has a job but its cache is dirty, clear its cache
- If we don’t know what a builder is doing (this is actually important when we start up this AI in the middle of a build because the last version of the AI crashed
), ask it - If a builder is idle, give it the next build job on the list
- If a builder is idle and we don’t have any build jobs on the list and there might be more jobs that are ready to start, give it a special job to figure out what new build jobs are ready to start
- If a builder is idle and none of those above conditions apply, let it go to sleep