Get_my_glitch_stuff.sh

[wh0]

suddenly it’s in demand to download a copy of your stuff on glitch. here’s a shell script that I’ve just used to do it myself. the situation being urgent enough to justify posting this without polishing it up please accept this under an MIT license

#!/bin/sh -eu
if ! type snail; then
	echo >&2 "snail not available, use 'npm i -g glitch-snail' to install"
	exit 1
fi
if [ ! -e projects.txt ]; then
	snail project list >projects.txt.partial
	mv projects.txt.partial projects.txt
fi
project_domains() {
	tail -n+2 projects.txt | cut -d' ' -f1
}
project_domains | while read -r domain; do
	if [ -e "projects/$domain/app.tar.gz" ]; then
		continue
	fi
	echo >&2 "downloading $domain"
	mkdir -p "projects/$domain"
	snail download -p "$domain" -o "projects/$domain/app.tar.gz.partial"
	mv "projects/$domain/app.tar.gz.partial" "projects/$domain/app.tar.gz"
done
if [ ! -e glitch_assets_urls.txt ]; then
	project_domains | while read -r domain; do
		if [ -e "projects/$domain/glitch_assets.jsonl" ]; then
			continue
		fi
		echo >&2 "extracting glitch assets list from $domain"
		tar -xf "projects/$domain/app.tar.gz" -O "app/.glitch-assets" >"projects/$domain/glitch_assets.jsonl.partial" || true
		mv "projects/$domain/glitch_assets.jsonl.partial" "projects/$domain/glitch_assets.jsonl"
	done
	if ! type jq; then
		echo >&2 "jq not available"
		exit 1
	fi
	echo >&2 "combining assets lists"
	project_domains | while read -r domain; do
		cat "projects/$domain/glitch_assets.jsonl"
	done | \
		jq -r '.url | select(.)' | \
		sort | \
		uniq | \
		sed -e 's/%2F/\//g' -e 's/%3A/:/g' \
		>glitch_assets_urls.txt.partial
	mv glitch_assets_urls.txt.partial glitch_assets_urls.txt
fi
if ! type wget; then
	echo >&2 "wget not available"
	exit 1
fi
echo >&2 "downloading assets"
wget -c -i glitch_assets_urls.txt -x -P assets

dependencies:

• https://snail-cli.glitch.me/
• jq
• wget

saves project tarball (should be equivalent to clicking “Download” on dashboard) and assets

stuff that this should save (please reply if any of this doesn’t work):

• project disk, which includes code, gitignored stuff, rewind history (as .git), .env, .data
• assets
  • caveat: assets are requested without cache-busting url parameter and may be stale if you recently updated assets.
  • more caveat: asset list processing is kind of dumb and may end up downloading assets that you previously deleted
  • implementation note: asset urls contain %2F (an escaped /) separating project ID from asset name; this tool will convert that to a / so that projects’ assets are group into directories

notable stuff not saved:

• project description (although one line of it is retained in projects.txt)
• other project metadata, e.g. ID, collaborators, boost setting, remix history, created/modified date, legacy custom domain configuration
• anything you had in node_modules
• asset thumbnails

p.s. this is a no-fun zone and I will not be mailing you stickers for using this tool

p.p.s. ok medium-fun zone, I will add a counter here. reply and I’ll add your info too

569 projects, 5,389,724 KiB gotten

data

username	projects	KiB
wh0	336	3456812
DerDer56	83	368368
Pufferfish101007	150	1564544
total	569	5389724

[snotskie]

does what it says on the can

had to tweak to use npx snail instead, but otherwise no fuss

[DerDer56]

83 projects, 368368 KiB, thanks

[MilesWK]

Having some trouble getting this going. Can you share the project that you did this with?

[snotskie]

here’s what I did

npm install glitch-snail

Followed the instructions to auth that tool on https://snail-cli.glitch.me/ :

1. Request an email with the code from Glitch: The friendly community where everyone builds the web. Don’t click the link in the email.
2. Copy the code from that email
3. snail auth code xxxxxxxxx-xxxxx-xxxxxxxx

Then to get wh0’s script to work, I deleted the first if block, then changed snail to npx snail everywhere

[MilesWK]

And did you run it on the snail project, or did you do it locally? Why am I having soooo much trouble with this?!

[snotskie]

oh, i ran it locally!

[MilesWK]

Ahh darn. Okay, I need to figure out how to get NPM on my computer!

[wh0]

oh right, run this from somewhere other than glitch, otherwise you’re kind of transferring everything from glitch back to glitch, which is kinda goofy

[MilesWK]

This is a powershell version that ChatGPT created (just gave it the old code, and copied the new). It might not work at all:

# Requires PowerShell 5.0+

# Check if snail (via npx) is available
if (-not (Get-Command "npx" -ErrorAction SilentlyContinue)) {
    Write-Error "npx not available, install Node.js and use 'npm i -g glitch-snail'"
    exit 1
}
if (-not (npx snail -v)) {
    Write-Error "snail not available, use 'npm i -g glitch-snail' to install"
    exit 1
}

# Create projects.txt if it doesn't exist
if (-not (Test-Path "projects.txt")) {
    npx snail project list | Out-File -Encoding utf8 "projects.txt.partial"
    Move-Item "projects.txt.partial" "projects.txt"
}

function Get-ProjectDomains {
    Get-Content "projects.txt" | Select-Object -Skip 1 | ForEach-Object { ($_ -split '\s+')[0] }
}

foreach ($domain in Get-ProjectDomains) {
    $archivePath = "projects/$domain/app.tar.gz"
    if (-not (Test-Path $archivePath)) {
        Write-Error "downloading $domain"
        New-Item -ItemType Directory -Force -Path "projects/$domain" | Out-Null
        npx snail download -p $domain -o "$archivePath.partial"
        Move-Item "$archivePath.partial" $archivePath
    }
}

if (-not (Test-Path "glitch_assets_urls.txt")) {
    foreach ($domain in Get-ProjectDomains) {
        $assetsFile = "projects/$domain/glitch_assets.jsonl"
        if (-not (Test-Path $assetsFile)) {
            Write-Error "extracting glitch assets list from $domain"
            try {
                tar -xf "projects/$domain/app.tar.gz" -O "app/.glitch-assets" | Out-File "$assetsFile.partial"
                Move-Item "$assetsFile.partial" $assetsFile
            } catch {
                Write-Warning "Could not extract assets from $domain"
            }
        }
    }

    if (-not (Get-Command "jq" -ErrorAction SilentlyContinue)) {
        Write-Error "jq not available"
        exit 1
    }

    Write-Error "combining assets lists"
    Get-ProjectDomains | ForEach-Object {
        Get-Content "projects/$_/glitch_assets.jsonl"
    } | & jq -r '.url | select(.)' | Sort-Object -Unique |
        ForEach-Object { $_ -replace '%2F','/' -replace '%3A',':' } |
        Out-File -Encoding utf8 "glitch_assets_urls.txt.partial"

    Move-Item "glitch_assets_urls.txt.partial" "glitch_assets_urls.txt"
}

if (-not (Get-Command "wget" -ErrorAction SilentlyContinue)) {
    Write-Error "wget not available"
    exit 1
}

Write-Error "downloading assets"
wget -c -i glitch_assets_urls.txt -x -P assets

[MilesWK]

For some reason, it fails and can’t find jq while reading some of the asset lists. I’m not bothered by it. I don’t need the assets. Just the code.

[wh0]

how does jq usually install on windows? or is there something powershell can do natively to extract the asset urls from the glitch assets file?

[MilesWK]

Dunno. Chat made the whole thing!

[Pomax]

Probably also of interest, Option to download all projects - #12 by Pomax

[Pufferfish101007]

downloaded 150 projects, 1564544KiB.
As reported in the other thread, cdn.hyperdev.com’s certificate has expired, but i can just rerun the asset bit with --no-check-certificate if I need any of those assets. However, there were also a couple of assets located at https://s3.amazonaws.com/hyperweb-editor-assets/us-east-1:ID/file.extension which just gave a 403. I suspect those won’t be any assets that I’ll care about, but it’s worth noting.
now i need to go through and unzip (untar? extract?) all of the archives… maybe I should have used @Pomax’s script

[Pomax]

No need for “should have”: you can just give my script a shot in a completely different dir and see if that leads to less worries. Trying one does not preclude trying the other =D

[MilesWK]

For some reason, mine couldn’t find jq when finding assets.

I didn’t know @pomax had a script. Where is it?

[Pomax]

I mentioned it in the bulk download thread, linked to a few comments up.

[wh0]

I personally never uploaded any assets during the hyperdev era. and if you didn’t either, it could be references copied over from starter projects.

from what I’ve seen from the asset cdn, it’s stored in various Amazon S3 buckets. I’m assuming there’s a way to request them from S3 using some other S3 API where you pass a bucket name and an object key, so it might be possible to get these theoretically without ignoring the TLS certificate check.

[wh0]

a few extra notes on usage:

1. if the script fails due to an intermittent problem e.g. internet connection loss or temporary server unavailability or missing dependency that you later installed, you should be able to rerun the script, and it’ll pick up where it left off
2. if it really can’t download a project e.g. with repeated HTTP 500 error, you can edit the projects.txt file that it makes and delete the line for that project. then it should skip over it