last thread: Actually building Nix packages on Glitch
Project URL: Glitch :・゚✧
Prologue
For a number of nights now, work has been proceeding in order to bring efficiency to the crudely conceived idea of a Nix package installer that would not only supply binary cached archives to the Nix store for use in newer projects but would also be capable of automatically running without any dependencies not already in the project container.
Such a package installer is NarFlinger (provisional name).
Now basically the only new principle involved is that instead of archives being downloaded by the execution of an already-installed Nix package, it is retrieved by the concurrent interaction of Node.js on stream pipelines.
The original program had an import of the .nar parser from my nix-cache-view project adapted to write to disk in such a way that functions for generating the archived files’ contents were in a direct pipeline with a Writeable stream.
The latter consisted simply of a number of require('fs').createWriteStream
s so fitted to the segment of the decompressing .nar that whole-package-buffering was effectively prevented.
The main Readable stream was of the normal HTTP IncomingMessage
type placed in a pipeline with the stdin of a child process running xz -d
,
every exit
event being connected by a .once()
to a Promise’s reject
callback awaited at the tail end of the async generators.
NarFlinger has now reached a high level of development and it’s being successfully used in the installation of Emacs 28.2.
Moreover, whenever an additional package installation is required, it can also be employed in conjunction with a previously installed Nix store directory to reduce shared dependency re-downloading.
No actual content about optimization today. Or rather, this thread isn’t really going to be about optimization, it’s more of just a sad story about Node.js.
In the next post, I’ll be describing the components of the full installer and how I condensed it into a simpler script to help focus on why it was running so slowly.