Anti-gallery post: gentoo and homebrew

anti-gallery post: there’s nothing presentable to show off. this is only a technical report on some stuff that hasn’t so far worked out. enjoy :person_tipping_hand:

Branching out from Nix

Nix is undergoing some stabilization for their upcoming 23.11 stable release. While I wait for that, I’m taking some time to investigate some other solutions for getting software on Glitch. Here are two mainstream solutions for “install stuff without being the system package manager.”

  1. Gentoo “Prefix” Project:Prefix - Gentoo wiki - Gentoo being a distro notable for compiling things from source with a lot of customizations available at package build time, it makes poetic sense that a team has developed a way to make the packages run from a custom location in the file system. And in fact, this Prefix project is reportedly maintained.
  2. Homebrew - Mac users would have long known about this, a package manager that uh… is not part of the operating system. I mean, that’s its whole subtitle, “The Missing Package Manager for macOS (or Linux),” that it didn’t come with the OS. And so it installs into its own directory and knows a good bit about how not to break OS-supplied tools.

In the posts that follow, I’ll look at these two in detail, focusing on:

  • how far I got in using them on Glitch
  • how they compare to Nix

Gentoo Prefix

This was my first time seriously using Portage, their package manager. And although that’s a personal note in what’s meant to be a technical report, you can understand it to mean that the answer to many implicit “why did you do that” questions that come up below is “because I literally didn’t know the right way to do it.”

Getting it on Glitch: patches

It definitely fails to bootstrap using their provided script, because it sets up a modern glibc, which doesn’t work in our project containers due to the kernel giving our container EPERM on newer syscalls Observations on future-readiness, Things encountered while building nixpkgs 22.05. We need those patches for faccessat2 (programs appear not to have permission to do things, or files appear not to exist) and clone3 (threaded programs fail, and indeed Gentoo bothers to use multithreaded xz to unpack source tarballs).

Comparison with Nix: adding patches

Gentoo makes it much easier to direct the package manager to add some custom patches to a given package. It’s easier both in terms of the number of steps and the less quantifiable amount that you need to understand fixed points in lambda calculus. In Gentoo, you make a directory in your /etc/portage/patches (or prefixed location as we’re doing here) named after the package name, and you put patch files for that package in that directory.

You’ll be done in the time it took you to find the circular flowchart on the Nix documentation Overlays - NixOS Wiki that makes you realize you knew nothing about functional programming.

Getting it on Glitch: saving compiled things

The other usual thing I need is a way to save my work and resume it, because you only get 12 hours between /tmp wipes (less if you’re unlucky and your project starts in a container with higher uptime (or is uptime rather a property of the host server?)), and even within those 12 hours, it’s kinda hard to make sure you don’t forget and close your browser.

The Gentoo bootstrap process is split up into 3 stages.

  1. Stage 1 gets some software without the help of a package manager. Think raw wget, ./configure, and make with your host compiler. It sets up the package manager (Portage) and its dependencies.
  2. Stage 2 uses the package manager from stage 1 to build a good enough compiler, using your host compiler, to use in the rest of the boostrap process.
  3. Stage 3 uses the compiler from stage 2 to build the real compiler and real package manager and then uses those to build and install the base Gentoo system.

Stage 1 is comparatively quick. In my opinion, you can just get through it in one sitting. It produces some stuff in a temporary area that won’t be included in the final bootstrapped system. You can just archive this directory up into your ~/.data directory. A .tar.xz of it was only ~54 MB. I further moved it to assets to save space.

By the time you get to stage 2, you have Portage, so you can start using the built in support for binary packages Binary package guide - Gentoo wiki. To oversimplify countless pages of wiki stuff and manual pages:

  • Run emerge (that’s the command in Portage for installing a package) with the --buildpkg (-b) and --usepkg (-k) flags.
  • This will put built packages in a directory /var/cache/binpkgs (or prefixed location as we’re doing here) and look for them there to skip having to rebuild them.
  • Oh but the bootstrap script runs emerge for you, so actually put those buildpkg+usepkg flags into a config file somewhere.

And then you can sync that binpkgs directory into your ~/.data directory to persist it. When you configure Portage to use xz compression, the packages take ~82 MB. After stage 2 finished, I further archived them up and moved it to assets to save space.

And stage 3 is similar, because it’s just using Portage again. But it’s too big to fit into ~/.data. Using xz compression, the total size of all packages built in stage 3 came out to ~311 MB. At that point I had to build a script Assets 🔄 /tmp sync to sync this binpkgs directory with assets instead of ~/.data.

Comparison with Nix: how hacky is the bootstrap

Gentoo Prefix’s bootstrap does a lot to manage the different tools leading up to the final system. There’s all sorts of hacks built into the script to make each stage use the right toolchain and stuff. On the other hand, this kind of thing is Nix’s forte, with each build knowing exactly what program to use (except for a few weird cases).

The Gentoo Prefix bootstrap installs some preliminary tools built with your host compiler into (your-prefix)/tmp, while putting a couple of suites of more proper tools built with Gentoo’s own modern compiler into (your-prefix). This too is where Nix has it much easier. Nix natively keeps all builds separate, taking into account what tools were used to build each package.

But then, there’s this philosophical thing about Nix. You can never get to this state of nirvana where “the compiler that comes with this distribution can compile the distribution itself.” Because some hidden precursor compiler compiled at least the core build tools in that distribution. Thus you’d be using a different compiler in those cases, which would cause the resulting packages to have different hashes.

Comparison with Nix: less recompiling

Nix is very liberal about when it recompiles things. Bugfix in the compiler? Recompile all programs that you compiled with it. Minor change in a shared library? Recompile all dependents. Gentoo I believe lets you get away with much more changes without recompiling a package.

Getting it on Glitch: how to serve binpkgs

I don’t have a nice way to do this yet.

Binary package servers need to have a specific structure Binary package guide - Gentoo wiki, where there’s a “Packages” file at the top level that lists everything it has, and then binary package files organized into directories. That is:

  • There’s this Packages file that needs to be updated every time you add a new binary package.
  • There’s the actual binary packages, which are big.
  • And these had better be on one host in the specified organization.

That right there is kind of an unwholesome combination for Glitch, because:

  • Put it on assets → you can’t update the Packages file because it’ll get cached for too long.
  • Put it on a static site → you run out of room to put the actual binary packages because you’re limited to 200 MB.
  • Put binary packages on assets and the Packages file on a static site → assets are in a separate domain, so you’ll break the organization that Portage needs.
  • Redirect from the project to the assets → you can’t make actual HTTP redirects in a static site.
  • Use a full stack site → you’re limited to whatever few requests per hour, and you use up project hours.

Comparison with Nix: binary package servers

A nice thing about Gentoo is that there’s no rule that you have to serve the whole dependency tree for any packages that you have on your server. Nix requires that.

A nice thing about Nix is that you don’t need a central “Packages” index file. Nix can look up the package you need by a hash of all the source code, dependencies, and settings used to compile the binary package that it’s looking for. Gentoo needs to look through the package index to find one that matches the name, version, and compilation settings. You can just add built packages and not edit anything on the way.

Comparison with Nix: what counts as a dependency

Gentoo has this very detailed specification Package Manager Specification of what environment Portage builds run in. And anything installed in order to meet that specification—build tools, system utilities, common libraries—doesn’t count as a dependency. Nix on the other hand is much more explicit: assume basically nothing else exists other than your dependencies.

The result of this is that the minimal installation of Gentoo is quite some larger than Portage itself. You need a decent suite of compiler tools just to meet the Package Manager Specification. The minimal installation of Gentoo that comes from the Prefix bootstrap is ~264 MB (compressed with xz). The minimal Nix, being the package manager itself, is ~31 MB (compressed with gzip, so not exactly apples-to-apples).

Progress report

With the patches, the bootstrap works, and it gets you to a point where you can use the bootstrapped system. And with the binary packages synced to assets, a solo developer can build packages incrementally.

But I don’t have a good sense of what to do with the resulting bootstrapped system. It’s kinda big.

And I don’t have a way set up Glitch to distribute binary packages.

And, something I didn’t even mention, the current Gentoo distribution is kinda broken right now, with a dependency loop between curl and nghttp2 901101 – Running stage3 failed *Error: circular dependencies net-misc/curl-7.88.1-r1 (Change USE: -http2), so whatever normal emerge (something) commands I tried would just error out. They hack in an export USE="... -http2" in the bootstrap script, but after the bootstrapping you’re on your own. Am I reading this right? Is this something Gentoo users just have had to deal with since March? Does non-Prefix Gentoo somehow not have this problem? How? Aaaaaaaah!

So anyway, I gave up for now.

You’re-cool-if-you-care-about-this-stuff-but-please note: sizes in this post are measured in MB (10^6 bytes), not MiB (2^20 bytes).

I’ll post again about Homebrew. I’ll just tell you in advance: it’ll be less than this.

edit: Glitch :・゚✧ here’s the project, btw. it’s just whatever state it was in when I gave up.


interim post:

turns out you can run USE='-http2' emerge dev-util/cmake to tell Portage to use the non-http2 copy of curl from the bootstrap to build cmake. it doesn’t produce some kind of non-http2 cmake. there’s no http2 functionality in cmake to begin with. then with that perfectly normal cmake, you can build nghttp2 and the yes-http2 curl emerge net-misc/curl. and then you can install stuff normally.

Portage simply understands that “this is cmake with all the usual features” keeping no record of “you built this with a non-http2 curl” the way Nix would.

with that out of the way, I’m now [1225/1672] on the way to compiling Node.js 20 :raised_hands:

1 Like


You know, I know some people who use Macs. And recently I asked “what’s that thing you use to install stuff?” “Homebrew.” “Ah yeah. So Homebrew? Not Brew?” “… Homebrew.” And after a few minutes of looking into it, I finally understand: brew is the command, Homebrew is the name of the project. Which turned out to be just the beginning of learning new terminology. Right smack dab at the top of the manual for brew is a great big terminology section telling you that they insist on calling things formula, keg, rack, keg-only, Cellar, tap, and bottle, among other themed names.

Which hey, if giving every concept a themed term is the weirdest thing about a package manager, then that’s pretty good. I gotta say, from what I’ve seen, it is. At every turn, Homebrew seems to have a very normal way to do just about everything.

Getting it on Glitch: breaking the rules right out of the gate

Homebrew supports Linux. Parenthetically if you read only as far as the second line on their website, but no, they do.

They need root access in order to create their “Cellar,” their place where they store the built packages, in /home/linuxbrew (note to people keeping track: /home/linuxbrew is not the Cellar; the Cellar is further within that directory). Or you “may” install it to a different prefix. “May” is qualified from this on their page about using Homebrew on Linux Homebrew on Linux — Homebrew Documentation :

… you shouldn’t …
… buggy and unsupported …
If you decide to use another prefix: don’t open any issues, even if you think they are unrelated to your prefix choice. They will be closed without response.

So I’ve been very quiet about things, and I’ll only be talking about it here.

Getting it on Glitch: breaking more rules

From their installation documentation Installation — Homebrew Documentation :

Make sure you avoid installing into:

  • /tmp subdirectories because Homebrew gets upset.

We don’t want to waste a bunch of space in /app for what are essentially dependencies. Upset is better than nonexistent, right?

/tmp/homebrew seems reasonable. Note: if you’re stopping reading here, don’t use /tmp/homebrew. There’s a note about this later.

Getting it on Glitch: patches

Homebrew on Linux installs its own glibc, which we need to patch with our EPERM fixes. And they have a way to add custom patches: you have to edit the “formula” file, which is a description of how to build a package, adding a statement to it:

  patch do
    url "https://.../glibc-rhbz1869030-faccessat2-eperm.patch"
    sha256 "ed...41"

Comparison with Nix: adding patches

At some level, it’s the same as Nix. You need to add an instruction to apply a patch in the build instructions. Both have that file with build instructions in enormous Git repos. In Homebrew, that repo is called a “tap,” while in Nix, that repo is called a “flake.”

But in Nix, you have an alternative to forking the flake repo: you can configure an “overlay” to say “instead of the list of patches from the flake, use the list of patches plus this other patch of mine.”

In Homebrew on the other hand, what seems to work is to make a copy of the file from that huge tap repo into a new tap. And your formula will have the same name, which will allow you to have your glibc used in place of the glibc from the main homebrew/core tap.

Getting it on Glitch: what just happened to my Homebrew installation

It turns out that the build procedure for glibc uses specifically /tmp/homebrew to unpack some temporary bootstrapping tools, and it deletes it after the build finishes. We’ll need to use a different directory to install Homebrew itself. I’m going with /tmp/h then :roll_eyes:

Getting it on Glitch: saving compiled things

So I built the custom glibc implementation. Homebrew calls prebuilt binary packages “bottles,” and brew has a suite of commands for producing them. Odd thing: they insist on having their own distribution of tar for creating these bottles, while they’re pretty happy using something else to install from bottles. Oh they have a term for installing a package from a bottle, too. “Pouring.” You’re welcome.

Comparison with Nix: binary package compression

Homebrew uses gzip to compress bottles, for it speed. Nix has been using xz, which we earlier found out Speeding up a custom-built Nix package installer - #4 by wh0 its decompression was CPU-intensive enough to take longer than the download. So maybe gzip is a good thing on Glitch.

Comparison with Nix: binary package contents

Nix is much stricter about a package simply being a tree of files. Homebrew allows packages to have “post-install” steps, which may modify the package. Or to be more specific, the “keg.” That’s what they call an installed package.

The effect of this is that in Homebrew, you need to install a package in a “build-bottle” mode which defers the post-install steps in order to preserve a pristine keg to make a bottle. Then you run a command to create the bottle, and then you run a command to run the deferred post-install steps.

Nix doesn’t have this distinction. Rather, all the files have to be completely done after building, and they’re stored read-only in the Nix store. You can copy them into a binary cache whenever.

Comparison with Nix: installing binary packages

This bit about Homebrew is a little too defaults-focused for me. In Homebrew, information about where to find the bottle for a given package is stored in the formula. So you can’t go and build a bunch of bottles for a non-default Cellar and expect brew to install them automatically unless you also fork the whole tap and edit every formula you build.

brew can install single bottles from a filename, but it’s many times more fiddly when you have to deal with dependencies.

In Nix, the binary cache configuration is decoupled from the flake and derivations. You can set something in a config file to tell Nix where to get binary packages.

Getting it on Glitch: progress on bootstrapping

As the documentation had forewarned, getting Homebrew to set up a suite of packages in a different prefix was buggy. I got about as far as failing to compile krb5, a library deep, deep in the dependency graph on the way to installing node@20.

But secretly, I think it wouldn’t have worked even if I somehow had it installed in the normal prefix. There’s awkward stuff from the host system that leaks into the build. I think the subtext of “non-default Cellar directories don’t work” is that Homebrew significantly depends on bottles being built on CI servers that natively have the same glibc as the version in the distribution. Homebrew is not a good platform for building software from source on a host with older system software.

Comparison with Nix: relocatable bottles

Whereas in Nix fully treats the contents of a package as fixed, Homebrew has a system for patching packages at install time to fill in paths based on where they’re installed, including to non-default Cellar directories. That’s pretty darned merciful. I compiled gcc a total of zero times while working with Homebrew :pray:

Getting it on Glitch: breaking more rules

There’s a system that’s supposed to determine automatically if bottles are relocatable. You can override that decision with a --force-bottle when you run brew install. The packages I tried turned out to work enough in this mode.

Progress report

I then tried installing everything but glibc with --force-bottle, and seem to be able to run node@20 okay. Installing one of these “@version” things is what’s called “keg-only,” which means that brew won’t link it into your path, so you have to run /tmp/h/opt/node@20/bin/node. Their npm distribution uses #!/usr/bin/env node though, so you have to run /tmp/h/opt/node@20/bin/node /tmp/h/opt/node@20/bin/npm instead :skull:

Using Homebrew to set up Node.js 20 is kind of slow though.

cd /tmp
mkdir /tmp/h
curl -L | tar -xz --strip 1 -C /tmp/h

mkdir -p /tmp/homebrew-cache
ln -snf /tmp/homebrew-cache ~/.cache/Homebrew

mkdir -p /tmp/h/Library/Taps/wh0
ln -sf /app/Taps/wh0/homebrew-glitch -t /tmp/h/Library/Taps/wh0

eval "$(/tmp/h/bin/brew shellenv)"

brew install ~/.data/glibc--*.tar.gz
brew deps -n node@20 | xargs -n1 brew install --force-bottle
brew install --force-bottle node@20
$ time sh -eux
real    7m2.065s
user    3m10.872s
sys     0m44.692s

We have to work around limitations in brew install not recursively respecting --force-bottle (seems to be tracked? --force-bottle needs to be made recursive · Homebrew · Discussion #4875 · GitHub) which I’m doing with a lot of repeated invocations of brew, so we’re incurring several Ruby startups :weary:

and as usual, Glitch :・゚✧ here’s the project in whatever state it is in. that glibc bottle I copied to Glitch assets too

1 Like