NAR Flinger, a package installer in a single script

[wh0]

last thread: Speeding up a custom-built Nix package installer

Project URL:

• first sample project narflinger-nodejs-18
• installer development narflinger

I’ve been using the Nix package manager to build a suite of newer software for Glitch for a while. The hope has been to use the packages built that way in projects that want a newer version of something, such as a language runtime. But it’s been complicated to set up Nix in order to install those packages.

In this post, I present a simplified installer script, NAR Flinger, for setting up packages from a binary cache. NAR Flinger avoids several sources of friction that Nix encounters when running on Glitch.

Background: Nix the proper way

Setting up and using Nix on Glitch complicated. Here are several of those complications to illustrate why.

Nix itself isn’t available in our project containers, so you’d install that.

First is getting the binaries. I have a great big tarball with Nix and all its dependencies. Incidentally, that’s served as a tarball of a Nix “store,” which means unpacking it sets it up as if you had used Nix to install Nix. And that’s quite big, so you’d want to install it into /tmp. And /tmp is ephemeral, so you have to write something in your project startup to do it. But also not to do it if Nix is already installed. Complicated.

Second, Nix operates on /nix/store by default, which doesn’t exist in project containers, and we’re not permitted to create it, so you’d set up some environment variables to have it work somewhere else, /tmp/nix/store in this case. In Glitch, the normal way to do that is to write something in your .env file. But that means it doesn’t get copied over to remixes, and people viewing your code can’t see what values you’ve put. Complicated.

Third, Nix downloads prebuilt packages from a “substituter,” which can be a “binary cache.” But the binary cache that it uses by default only has packages compiled to be used from /nix/store, which you can’t use in a Glitch project container, so you need to set it up with a different substituter and the public key for that substituter. The normal way to do that in Nix is to create a config file, which goes in /app/.config/nix/nix.conf. But on Glitch, /app/.config is in the global gitignore, so it won’t show up in the editor. Complicated.

Fourth, the Nix program is in some gnarly path in /tmp/nix/store/somecryptographichash-nix-whateverversion/bin, which you wouldn’t want to type out, so you need to do something to get that into your PATH. The normal way to do this in Nix is to have a “profile,” which collects a bunch of symbolic links in a relative stable location, /tmp/nix/var/nix/profiles/per-user/app/profile/bin, and add that to your path. You can use Nix itself to set up the profile, so you’d just run this one command from the complicated /tmp/nix/store/… path. But of course since the profile is in /tmp, you gotta set something up to do that if it doesn’t exist. Complicated.

Fifth, it turns out the more normal way is not actually to put /tmp/nix/var/nix/profiles/per-user/app/profile/bin in your path, but to put a shell script provided by Nix into your .bashrc, which does the PATH setup for you, so you somehow do that. But Glitch doesn’t run your .bashrc when it executes your project’s “install” and “start” scripts in your package.json. So you need to arrange for it to run explicitly. Possibly by adding it to the beginning of your install and start scripts. Oh but that shell script itself, it’s installed as part of Nix. So it’s not going to exist when your project starts up. Make sure to write your install script in such a way that it doesn’t crash when Glitch runs it before you install Nix. Complicated.

Then you’ll have the Nix package manager set up, but you haven’t actually installed the thing you wanted yet.

Sixth, you’d use Nix actually to install something. And I’ve seen people recommend various packages to do this declaratively. Nix does have a native way where you run nix-env -i to install something, which I think works too. So you can go ahead and put a bunch of those into your “install” script. Not too complicated, but kind of ugly to write all those commands in a JSON string.

NAR Flinger

NAR Flinger (I’ve changed the capitalization and spacing since the provisional name in the last thread), in contrast, is a single script (currently written in Python), ~9 KB at the time of writing, that you configure in your package.json file. It’s small enough that you can just put it in your /app directory. And the configuration in package.json makes it easy to carry the configuration across project remixes.

Here’s how to configure it. From the sample project:

// package.json
{
  "scripts": {
    "install": "python3 narflinger.py", // (1)
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.18.2"
  },
  "narflinger": {
    "basenames": [ // (2)
      "r9gqywii82drj1m893kdsdkidc80ir9z-nodejs-18.14.1"
    ]
  }
}

1. You set an install script to run NAR Flinger.
2. You put a list of packages to install.

Sample project

Here’s a sample that uses Node.js 18’s built-in fetch to check if glitch.me appears on the Hacker News front page.

const express = require('express');

const app = express();
app.get('/', async (req, res, next) => {
  let result;
  try {
    const hnRes = await fetch('https://news.ycombinator.com/');
    if (!hnRes.ok) throw new Error(`Hacker News response ${hnRes.status} not ok`);
    const hnText = await hnRes.text();
    result = hnText.indexOf('glitch.me') !== -1;
  } catch (e) {
    next(e);
    return;
  }
  res.set('Content-Type', 'text/plain');
  res.send(`Is glitch.me on the Hacker News front page? ${result ? 'yes' : 'no'}`)
});
app.listen(process.env.PORT, () => {
  console.error('listening on port', process.env.PORT);
});

See if it is: https://narflinger-nodejs-18.glitch.me/

---

In the next post, I’ll talk about what NAR Flinger doesn’t support and walk through what the script does.

[_tr]

This is actually pretty cool.

[wh0]

Good things from Nix that NAR Flinger doesn’t do

• Nix maintains a database of what’s installed, including what depends on what. And that’s great if you later need to uninstall things. For our use case though, we’re installing into /tmp, which will be cleared each time Glitch recreates your project container. So things won’t be installed for more than 12 hours anyway. Although unfortunately, this makes NAR Flinger not actually compatible with Nix.
• Nix builds “profiles,” as mentioned above, which aggregate a bunch of symlinks into a directory structure. Importantly, it’s not just for the binaries. You also get symlinks to, for example, manpages. NAR Flinger so far only creates the symlinks for binaries. I’m interested in expanding this to other kinds of files.
• Nix can evaluate the Nix programming language, which is involved in a very normal way Nix users specify which packages they want to install etc. Normally they specify an “attribute” name, in a great big dictionary of those names to package definitions. That great big dictionary is defined by a program in a functional programming language—the Nix language. And being able to refer to a package by that attribute name is what lets normal users say they want “nodejs” instead of something as detailed as “r9gqywii82drj1m893kdsdkidc80ir9z-nodejs-18.14.1” as in our sample app. It’s important that NAR Flinger doesn’t do this though, because that great big dictionary of package definitions includes some things that Glitch will suspend your project for. But I am interested in publishing a list of these package names that you can use in NAR Flinger.
• Nix can build packages from source. That’s useful for when you want to customize something about a package. For NAR Flinger, I felt it would slow down the project installation too much if we were to compile things from source. If you must use a customized package build them using the real Nix and upload them to a binary cache for NAR Flinger to install from. Actually the current suite of software available from my binary cache contains some patches to make modern glibc work on Glitch (read more).
• Nix performs integrity/authenticity checks on the packages you download from a binary cache. That’s why you set up a public key when using a custom binary cache containing packages built by someone other than the NixOS project. NAR Flinger doesn’t do these, so you’ll have to rely on the integrity of the binary cache host, which in this case is Glitch’s asset CDN and GitHub for packages that don’t fit on Glitch’s asset CDN (read more).
• Nix makes the “store” read-only. Which is fine, because in Nix you don’t edit or overwrite things. When you edit a package by changing the source or changing the dependencies or adding patches, Nix computes the updated cryptographic hash of all sources and instructions, resulting in a new location to store the results of the build. NAR Flinger doesn’t mark everything as read-only. There’s no special reason not to. Although I’ll note that it made resetting things during development simpler.

NAR Flinger internals

NAR Flinger by default uses /tmp/nix/store and my binary cache, which are suitable for use on Glitch. But you can specify narflinger.store_prefix and narflinger.store_prefix and narflinger.base to override these, respectively.

At the core of it is routines to crawl a package’s dependency tree in a binary cache and to parse the contents of a NAR file. These I ported from the Node.js version that was a precursor to the current Python implementation (read more). And that Node.js version I adapted from a script for web browsers (read more).

If the search for dependencies encounters a package that’s already in the store, it skips installing it and searching for dependencies of that package.

NAR Flinger collects the list of packages to install using a depth-first search, following each package’s dependencies. It visits these packages in post-order, so that it installs a package’s dependencies before installing the package. This helps keep the contents of the Nix store in a state that Nix calls “valid” (except for that part about the database, NAR Flinger doesn’t maintain a database), meaning that a package in the store has its dependencies in the store too.

When parsing through a NAR file, NAR Flinger writes its contents out to a temporary directory. When it finishes unpacking it, it renames the directory to its store path. That prevents it from creating any partial packages in the store.

After it unpacks a package that’s requested in the package.json, it links the the files in the package’s bin directory into your ~/.local/bin directory. Note that the files in the bin directories of dependencies don’t get linked this way, so if you want something make sure it’s explicitly listed in your package.json. If multiple packages listed in your package.json provide a given name in the bin directory, packages later in the list will win.

Weird thing when using this to install Node.js

When your project first starts up after Glitch sets up a new project container for it, Glitch uses the old built-in version of npm and Node.js to start the npm install command. So it’ll complain about the package-lock.json being in a future version that it doesn’t know about. Then the install script runs to install the new version of Node.js and npm.

If the install step runs again, for example if you edit your package.json, then the new version of npm and Node.js get used, which then complains about the old version of package-lock.json.

How might we work around this?

---

I’m going to try making some more projects to demo other modern runtimes. I’m have the packages listed in the NixOS “small” set built (read more), so there should be a good few to try out.

[wh0]

Sample project: Emacs

narflinger-emacs

This project installs Emacs. I’m so amused with the built-in web browser. (Edit: I’m aware that project containers come with Emacs preinstalled. The provided versions predates the addition of this web browser though.)

That’s us!

If you want to play around with it, here’s how:

1. Remix that project.
2. Open the terminal.
3. Type emacs and press enter.
4. Press Alt+X, type eww, and press enter.
5. Enter a URL. You’ll have to figure out what sites work well in text mode and without JavaScript though.
6. Move around with the arrow keys/Page Up/Page Down/other Emacs bindings if you’re familiar with them/Tab/Shift+Tab
7. Press enter on a link to open it.
8. Press L to go back and R to go forward.
9. Press G to enter another URL.
10. Press Ctrl+X, Ctrl+C to exit.

Oh and the dunnet game is pretty fun too. Make sure you keep the Glitch editor open in the background, or they’ll shut down the project container and you’ll have to start over.

[doxr]

This thing is cool I will say it’s pretty big

[wh0]

Sample project: Python 3.10

narflinger-python-310

If you want a Python newer than the 3.7 that comes with the project container, here you go. Now you can install things that require newer Python and use new language and standard library features.

Here I’m using the new match syntax from Python 3.10:

import math

import flask

app = flask.Flask(__name__)

@app.route('/')
def index():
  return flask.send_file('index.html')

@app.route('/test/<command>')
def test(command):
  # check this out, match!
  # https://peps.python.org/pep-0636/
  match command.split('-'):
    case ['add', a, b]:
      return str(int(a) + int(b))
    case ['multiply', a, b]:
      return str(int(a) * int(b))
    case ['ln', x]:
      return str(math.log(float(x)))
    case _:
      flask.abort(400)

[wh0]

Sample project: PostgreSQL

narflinger-postgresql

Much respect to SQLite, but if you want to familiarize yourself with the nuances of different SQL databases, here’s a project with PostgreSQL.

const express = require('express');
const pg = require('pg');

const client = (async () => {
  const c = new pg.Client({
    user: 'app',
    database: 'postgres',
    host: '/tmp/run/postgresql',
  });
  await c.connect();
  await c.query('CREATE TABLE IF NOT EXISTS poll (option INTEGER PRIMARY KEY, count INTEGER)');
  await c.query('INSERT INTO poll (option, count) VALUES (1, 0), (2, 0), (3, 0), (4, 0) ON CONFLICT DO NOTHING');
  console.error('setup db done'); // %%%
  return c;
})();
client.catch((e) => {
  console.error(e);
});

const app = express();
app.use(express.static('public'));
app.post('/vote/:option', (req, res, next) => {
  (async () => {
    try {
      const c = await client;
      const r = await c.query('UPDATE poll SET count = count + 1 WHERE option = $1::integer', [req.params.option]);
      if (r.rowCount === 0) throw new Error('update didn\'t affect any rows');
    } catch (e) {
      next(e);
      return;
    }
    res.redirect(302, '/');
  })();
});
app.get('/results', (req, res, next) => {
  (async () => {
    let r;
    try {
      const c = await client;
      r = await c.query('SELECT option, count FROM poll ORDER BY option');
    } catch (e) {
      next(e);
      return;
    }
    res.send(r.rows);
  })();
});
app.listen(process.env.PORT, () => {
  console.error('listening');
});

You can also vote on the sample poll here: https://narflinger-postgresql.glitch.me/

[edit 2023 may 20:

wow landslide]

In the process of setting this up, I found out that PostgreSQL doesn’t work when you symlink their binaries from somewhere else .

[wh0]

Sample project: PHP

narflinger-php

PHP is included in NixOS “small,” so here’s a project with that installed. It’s running with that development server without apache or nginx.

<?php
phpinfo();

I haven’t used PHP in a long time though, and I don’t have a fancy starter. Maybe there’s a framework that people like to use…

Sample project: Laravel

narflinger-php-laravel

The PHP installation seems to work well enough that Composer and Laravel install without too many complaints.

It looks like it stores some non-code things in /app outside of .data though. Maybe there’s a way to configure it to store it somewhere within .data. Anyone who’s familiar with Laravel, please advise.

[_tr]

Here’s something I noticed about the PHP project: the phpinfo() function gives out your whole .env, and that includes the project invite token, that’s exposed

[hello-smile6]

Could that environment variable be unset to prevent that?

[wh0]

oops yeah, people better not use phpinfo in their projects

[hello-smile6]

Doesn’t php-fpm unset environment variables by default?

[wh0]

three things:

1. does it? wasn’t aware that php-fpm did that
2. this is php -S. does that even go through CGI?
3. and if it did, then would people need a different way to get secrets from their .env file into their php scripts?

[hello-smile6]

@wh0

does it? wasn’t aware that php-fpm did that

Yes. I have at least 8 different values that shouldn’t be leaked in env variables on a server that runs lots of things (including php-fpm and nginx), and php-fpm hides all non-allowlisted env variables by default.

@wh0

does that even go through CGI?

No.

@wh0

and if it did, then would people need a different way to get secrets from their .env file into their php scripts?

If php-fpm was used, people could manually allowlist approved variables and strictly regulate the use of phpinfo().

[wh0]

what env vars do you use that shouldn’t be propagated to php by the way? why are they environment variables in that case? are they for another program to use?

[hello-smile6]

@wh0

what env vars do you use that shouldn’t be propagated to php by the way?

I had several TLS certificates, passwords, and other information in environment variables.

@wh0

why are they environment variables

Because I couldn’t store them any other way.

@wh0

are they for another program to use?

Yes.

[wh0]

@hello-smile6

Because I couldn’t store them any other way.

https://help.glitch.com/kb/article/18-adding-private-data/

you can put files in the .data directory too, if that’s any easier

@hello-smile6

I had several TLS certificates

woa what did you use these for? doesn’t Glitch do the TLS termination for you?

[hello-smile6]

@wh0

you can put files in the .data directory too, if that’s any easier

It wasn’t on Glitch

@wh0

doesn’t Glitch do the TLS termination for you?

That doesn’t help with TLS client certificates or when you tunnel TLS over a websocket for linking IRC servers to each other

[wh0]

Sample project: MariaDB

narflinger-mariadb

MariaDB probably isn’t as exciting, since the project containers already have MySQL, which is closely related. But still, here’s this.

  const c = await mariadb.createConnection({
    user: 'app',
    socketPath: '/run/mysqld/mysqld.sock',
  });
  await c.query('CREATE DATABASE IF NOT EXISTS ranking');
  await c.query('USE ranking');
  await c.query(`
    CREATE TABLE IF NOT EXISTS items (
      id INTEGER PRIMARY KEY,
      tier INTEGER
    )
  `);

It turns out that Glitch has quite some special stuff in the container to make MySQL work (and which also serves to make the highly MySQL-compatible MariaDB work). For example, /run/mysqld is owned by the app user. And /etc/mysql is symlinked to /app/.mysql. Huh.

[hello-smile6]

How do people find the package names?