suddenly it’s in demand to download a copy of your stuff on glitch. here’s a shell script that I’ve just used to do it myself. the situation being urgent enough to justify posting this without polishing it up please accept this under an MIT license
#!/bin/sh -eu
if ! type snail; then
echo >&2 "snail not available, use 'npm i -g glitch-snail' to install"
exit 1
fi
if [ ! -e projects.txt ]; then
snail project list >projects.txt.partial
mv projects.txt.partial projects.txt
fi
project_domains() {
tail -n+2 projects.txt | cut -d' ' -f1
}
project_domains | while read -r domain; do
if [ -e "projects/$domain/app.tar.gz" ]; then
continue
fi
echo >&2 "downloading $domain"
mkdir -p "projects/$domain"
snail download -p "$domain" -o "projects/$domain/app.tar.gz.partial"
mv "projects/$domain/app.tar.gz.partial" "projects/$domain/app.tar.gz"
done
if [ ! -e glitch_assets_urls.txt ]; then
project_domains | while read -r domain; do
if [ -e "projects/$domain/glitch_assets.jsonl" ]; then
continue
fi
echo >&2 "extracting glitch assets list from $domain"
tar -xf "projects/$domain/app.tar.gz" -O "app/.glitch-assets" >"projects/$domain/glitch_assets.jsonl.partial" || true
mv "projects/$domain/glitch_assets.jsonl.partial" "projects/$domain/glitch_assets.jsonl"
done
if ! type jq; then
echo >&2 "jq not available"
exit 1
fi
echo >&2 "combining assets lists"
project_domains | while read -r domain; do
cat "projects/$domain/glitch_assets.jsonl"
done | \
jq -r '.url | select(.)' | \
sort | \
uniq | \
sed -e 's/%2F/\//g' -e 's/%3A/:/g' \
>glitch_assets_urls.txt.partial
mv glitch_assets_urls.txt.partial glitch_assets_urls.txt
fi
if ! type wget; then
echo >&2 "wget not available"
exit 1
fi
echo >&2 "downloading assets"
wget -c -i glitch_assets_urls.txt -x -P assets
saves project tarball (should be equivalent to clicking “Download” on dashboard) and assets
stuff that this should save (please reply if any of this doesn’t work):
project disk, which includes code, gitignored stuff, rewind history (as .git), .env, .data
assets
caveat: assets are requested without cache-busting url parameter and may be stale if you recently updated assets.
more caveat: asset list processing is kind of dumb and may end up downloading assets that you previously deleted
implementation note: asset urls contain %2F (an escaped /) separating project ID from asset name; this tool will convert that to a / so that projects’ assets are group into directories
notable stuff not saved:
project description (although one line of it is retained in projects.txt)
other project metadata, e.g. ID, collaborators, boost setting, remix history, created/modified date, legacy custom domain configuration
anything you had in node_modules
asset thumbnails
p.s. this is a no-fun zone and I will not be mailing you stickers for using this tool
p.p.s. ok medium-fun zone, I will add a counter here. reply and I’ll add your info too
downloaded 150 projects, 1564544KiB.
As reported in the other thread, cdn.hyperdev.com’s certificate has expired, but i can just rerun the asset bit with --no-check-certificate if I need any of those assets. However, there were also a couple of assets located at https://s3.amazonaws.com/hyperweb-editor-assets/us-east-1:ID/file.extension which just gave a 403. I suspect those won’t be any assets that I’ll care about, but it’s worth noting.
now i need to go through and unzip (untar? extract?) all of the archives… maybe I should have used @Pomax’s script
No need for “should have”: you can just give my script a shot in a completely different dir and see if that leads to less worries. Trying one does not preclude trying the other =D
I personally never uploaded any assets during the hyperdev era. and if you didn’t either, it could be references copied over from starter projects.
from what I’ve seen from the asset cdn, it’s stored in various Amazon S3 buckets. I’m assuming there’s a way to request them from S3 using some other S3 API where you pass a bucket name and an object key, so it might be possible to get these theoretically without ignoring the TLS certificate check.
if the script fails due to an intermittent problem e.g. internet connection loss or temporary server unavailability or missing dependency that you later installed, you should be able to rerun the script, and it’ll pick up where it left off
if it really can’t download a project e.g. with repeated HTTP 500 error, you can edit the projects.txt file that it makes and delete the line for that project. then it should skip over it