Broadening Nix package builds to the NixOS "small" set

Last thread: Things encountered while building nixpkgs 22.05

Now that all these various incompatibilities with Glitch are patched up, I’ve been able to build several more packages other than the nix package manager itself. The target I’m going for is a slight modification of set of binaries built for the so-called “small” channel.

The modifications include:

  • drop certain system packages that we wouldn’t be able to use anyway, e.g. the Linux kernel
  • use non-GUI variants of packages, e.g. emacs-nox instead of emacs

The packages are (not the official package names, nix attribute name, official product name, or anything):

  • apache httpd
  • cmake
  • emacs
  • gettext
  • git
  • imagemagick
  • jdk
  • nginx
  • nodejs
  • openssh
  • php
  • postgresql
  • stdenv (apparently a collection of build scripts for nix? NixOS - Nix Pills)
  • subversion
  • vim

Their dependencies and build dependencies are included too.

Confusingly, the “small” channel itself contains just as many packages as the other channels. Rather, it’s differentiated in that the maintainers update it faster, and they only serve prebuilt binaries for a smaller set of packages. I’m actually actually still building from the stable channel (i.e. with the least frequent updates), thus the least impressive of both worlds.

Anyway, in scaling up the number of packages I build, I’ve encountered a few new issues.

Copying derivations uses too many HTTP requests

One of the earliest Glitch-specific setbacks I encountered in this project was that I couldn’t download the nixpkgs package index and compile packages in the same project. (See Install prebuilt packages without root, from nixpkgs - #3 by wh0 for info.) This involves copying “derivation” files (which contain a format of instructions for nix to download sources and compile packages) from one project to another. Nix has its own way to copy these files over HTTP, which I’ve been using.

But with the expanded set of packages to build, the increased number of derivation files is bumping up against Glitch’s hourly request limit. In my last build, there were 1,726 files needed for the complete build instructions. each file has its own metadata file that Nix needs to copy before it copies the file itself. I think Nix also does an HTTP HEAD request to make sure each file exists before trying to download it, although I haven’t looked into the details. But that conservatively means it would take 5,178 for Nix to copy over the build instructions, which is too many to do all at once. Glitch allows 4,000 requests per hour to a project.

Projects with unlimited requests bottlenecked nonetheless

I have a Glitch Pro membership for the time being, and it indicates that my projects–even unboosted–will be exempt from the request rate limit. However, attempting to copy the full build instructions results in HTTP 403s partway through. Materials I’ve found online say that you get a special HTTP 429 when you hit the request limit, so it must be some other effect.

This is actually a good problem to have though, as it’s my preference to build things that work on Glitch’s free plan. There’s plenty of problems that can be solved by buying boosted projects or even by buying a VPS. I’m just having fun solving them here on Glitch.

Generated static sites have trouble with thousands of small files

So I tried designing a new project to copy the derivation files from. Glitch’s generated static sites feature seem to be a good fit: I would generate the site by “instantiating” the derivations from nixpkgs, and then those derivations would be the files in a static site. Then that static site… somehow I’m under the impression that static sites and generated static sites get unlimited requests, but now that I really look, I can’t find that stated anywhere. Anyway, I tried this.

The new design would unfortunately mean no on-demand instantiation of derivations outside the “small” binaries set, but eh. I’ll just add other packages manually to the generation step if I want them.

The derivation files and their metadata files are pretty small. In all, they’re about 5.5 MB total. That fits easily in a project’s disk.

However, something on Glitch freezes up when it tries to shut down the project and to switch it over to static file serving. This results in not being able to access the site for some time, as well as not being able to connect to the editor, as well as all changes from the last editing session being reverted :grimacing:

I’m in contact with Glitch support about this, but there are no findings yet.

Static sites aren’t serving a certain file for some reason

Because using a generated static site wasn’t working, I also tried using a plain static site, where I would manually run the build process from the project terminal. Which is fine.

The protocol that Nix uses to copy stuff over HTTP involves first requesting a file called nix-cache-info, a text file with some metadata about the files available on the server.

Unfortunately, I ran into an issue where that file would disappear when the project container is off.

I’m in contact with Glitch support about this, but we haven’t been able to figure out why this is happening.

OpenJDK headless is too big to be an asset

It’s 316 MB, the max being 256 MB.

I’ve instead uploaded it to GitHub releases (max 2 GB) Release unversioned · wh0/tmpnix-oversized · GitHub and linked to it with a modified .narinfo metadata file https://cdn.glitch.me/35f4f809-f949-484d-af9c-269b3b7abc78/7mqh5i48y64kr1qvyzfb10cdvxsjliri.narinfo (see the absolute URL field).

So far I’ve done this manually. I guess I’ll have to have .narinfo rewriting in the uploader project after all, although it was really nice not having it up until now.

Moving to do .narinfo rewriting might also let us switch to using cdn.glitch.global for packages under 20 MiB. Although we don’t have a formal statement from Glitch that they’d prefer this.

5 Likes

ok the thing about that nix-cache-info file: seems it’s not being served because it doesn’t have a file extension. seems reasonable, as that leaves Glitch with no info about what content type to serve it with.

taking a page from the age old “how do I serve a page without the .html extension” question, I’m gonna move it to nix-cache-info/index.html. I think nix won’t mind if the server says it’s an HTML file

:person_facepalming:

3 Likes

addendum: I updated the asset uploader component to check the size and automatically (i) upload to github releases and (ii) rewrite the .narinfo to point to github releases. but I’m spiritually harmed by the loss of simplicity

4 Likes