Node versions on Glitch

Building Glitch
1

Hey all, Pomax of the engineering team here.

A bunch of folks have been (rightly) asking when Glitch’ll finally offer a current version of Node, because how hard can it possibly be to just update the version, and I’ll be the first to admit I’ve asked that same kind of question but things are rarely about “just” doing something and I’d like to explain why it looks like we’re doing nothing (when in fact we’re quite hard at work).

Some Details:

The main problem is an Operating System problem. Glitch projects are effectively virtual machines that are based on Ubuntu 16 and Linux kernel 4.4 - and those of you familiar with Linux might be raising an eyebrow at that because that version of Ubuntu is no longer supported by Canonical. And that’s really the source of why it looks like nothing is happening: we can’t just update Node because the current versions of Node need a version of glibc that isn’t available for Ubuntu 16, and even if it was, the current versions of Node need a version of Python (used when you need to install a node package that needs to be natively compiled) that isn’t just unavailable for Ubuntu 16 but used to be available and then got pulled from the package repository.

That leaves us in a position where we can’t really “just bump” packages up in version, and instead we’ve been stuck with having to essentially redo all the Glitch code relating to project hosting: creating VM images using a more modern version of Ubuntu so that we can even install the toolchains necessary for a project to install a current version of Node. And Python. And Rust. Etc. etc. etc.

How we’re working on this:

This is a relatively big undertaking, and it’s also a mostly invisible one: there’s no parts in that dev work that we can land early and go “look, we’re making progress”, we can’t land it until all the work is done, and we’ve confirmed that not just new projects, but also old projects that were backed up on Ubuntu 16 and then “woken up” on the new version work properly. There are a lot of subtle dependencies that we need to work through before we can consider that work a success, and unfortunately we often can’t give meaningful updates other than “turns out X also needs to be updated so that Y can be updated because that has a single Z that allows us to at least run this code path that we’re trying to uplift”.

And of course, this work takes a backseat when users report problems with their projects that we need to fix, or when someone decides that it would be hilarious to DDoS Glitch and distract the entire team for a few days to deal with the fallout. That work often takes precedence over long-term tasks like a complete backend uplift.

Where that leaves us:

While I’m sure this won’t satisfy folks: we are absolutely still working on this, we’re probably even more frustrated than you are that it’s taking so long (because we also can’t take advantage of all the modern features of Node on the dev side while we’re stuck on these old Ubuntu images), and we’re trying to get this pushed out as soon as we can.

13 Likes
2

thanks for the update @Pomax
happy dance

2 Likes
3

does glitch use anything not forwards-compatible from Ubuntu 16.04?

4

There’s a number of things that you’d expect to be forward-compatible but only work with peer dependencies (and peers of peers) that make it impossible to just grab the newer version on a newer ubuntu without also needing the newer versions of all of those, and then invariably some tool or config or compiled node module (and so far, Node itself a few times) relies on an ancient version of a peer and you’re back to “okay how do we bridge this gap” and sometimes that ends in “okay, this just doesn’t work, we’re going to have to rewrite things in 16.04 context first, to no longer depend on this one tool, and then retry the entire effort”.

5

can you post some, I’d like to see if I can help with this

6

As much as that’s appreciated, there are parts of the configuration that are core to the problem that you’re not going to run into with a stock 16.04 that we can’t make public but contribute to making this effort a lot harder than it would be if this was just a regular old Ubuntu VM or Container.

For example, you can (if you want to sit through that effort) compile a standalone version of Node 22 using a more modern versions of Ubuntu and then copy that into Ubuntu 16.04 and have that work “for the most part” (standalone unfortunately doesn’t actually mean standalone, there’s still some shared libs that Node refuses to compile into itself), or you can even take stock Ubuntu 16.04 and spend a few days manually compiling all the libs and tools necessary in user space and end up with a similarly mostly working version of Node, but dropping that same ELF into the image we’re forced to work with will result in an immediate execution error with a system level stack trace.

It makes you miss stock Ubuntu 16.04, which is an achievement =/

7

Sorry, I mean to ask about complications with this.

I just realized I missed this bit in your earlier reply indicating that you’re replying about something other than upgrading the project containers’ OS base. Personally, I stand with “Don’t Break Debian,” and I don’t encourage Glitch to copy libraries from other Ubuntu releases until Node.js 22 works :sweat_smile:

Side note, we in the community have done this and managed to patch things up not to crash on Glitch. I’d be happy to share my notes.

But let me put .so file melting pot ideas out of mind for a minute. Getting back to this:

I’d like to ask more specifically, what sort of things are making this complicated?

  • Is there a component that can create Ubuntu 16.04 containers but that can’t create containers that use newer releases of Ubuntu? (I’d believe it, btw)
  • Is there a toolchain Glitch uses that works in Ubuntu 16.04 but that didn’t go on to support newer releases of Ubuntu?
  • Is there something about Node.js that worked in Ubuntu 16.04 that no longer works in newer releases of Ubuntu?
  • Is there something about Python that worked in Ubuntu 16.04 that no longer works in newer releases of Ubuntu?
  • Is there something about Rust that worked in Ubuntu 16.04 that no longer works in newer releases of Ubuntu?
8

As much as I’d love to share details, going deeper into this would basically turn into a community development effort, and we don’t have the capacity for running that right now (we’re a small team atm), so even though it might feel like you could be able to help unblock some of the work if only you had some more details, vetting what details I can share and then turning that into a collaboration that can be signed off on (which would almost certainly include NDAs) is just not in the cards right now.

2 Likes
9

alright. thanks for replying to say so. I’m glad I had a chance to ask.

10

Thanks for the update! In case it helps, I leave you a gif:
image

2 Likes
11

Cool news, I hope you guys can figure out how to accomplish this. Keep it up!

Do you have an idea on what node.js version you guys will be upgrading to?

12

I feel this as working for a company that hosts on glitch and gets DDoS often and having to create and maintain the code to fix the vulnerabilities

14

It is 2025 now - still no progress?
NODE22 has LTS now - so please let’s skip upgrading one increment every 6 months and always be behind the crowds

15

what? someone ddosed glitch? doesn’t glitch have cloudflare?

16

It has several DDoS protection layers, but people are people and DDoS mitigation is not a one-and-done deal: there are enough bad actors always trying new things to cause headaches when you least need those.

2 Likes
17

Once everything is upgraded successfully, is there any chance that other runtimes like Deno and Bun get added or will Node.js stay as the sole runtime still?

18

Even if we don’t have prebuilt solutions, once the uplift is complete it should be a lot easier to just grab a binary for those and run it. We’re going to have to see where we’re at once the uplift is done, but “now we can finally add …” will be high on the list.

1 Like
19

what’s the uplift this time

20

I’ll qualify that we’re not currently planning on supporting Bun or Deno nor any explicit work to make them easier to run, but on a less decrepit operating system it will likely be easier to “hack” your own solution if we don’t package it in the environment.

There is no uplift from today’s maintenance window and unfortunately the next few steps aren’t likely to be particularly visible, but a meaningfully updated project container is significantly closer than it was when last we discussed it.

Cheers;
cori

21

let the guessing games begin

is it a container engine syscall allowlist update

edit: wait, maybe I’m reading that first paragraph wrong. “… we’re not currently planning on (x), but on (less decrepit operating system) …” :face_with_raised_eyebrow:

edit: it’s not :person_shrugging: