Assets 🔄 /tmp sync

The Gallery
1

Project URL: Glitch :･ﾟ✧

Alright picture this. It’s late night. Well it’s always late night when you’re on Glitch, but this is no exception. Your stuff’s been building all day. You’ve been copying your built packages into /app so that you don’t lose your progress. But you’re running out of project disk space. But they just keep coming. Compilers. Shared libraries. Language interpreters. Whatever the heck libksba is. More packages. And then… oh no. Another copy of GCC. It’ll be four hours down the drain if you don’t save this somewhere. But your Disk meter is at 187MB/199MB already. :cold_sweat:

But there’s plenty of space in Glitch Assets right? You just need…

  1. A convenient way to upload a directory of artifacts into the assets CDN
  2. A way to upload new artifacts when they’re added to that directory
  3. A convenient way to download those artifacts back into the directory later

So here’s asset-sync, a script—pair of scripts :person_shrugging:—to do that.

# put glitch-snail in your package.json
snail auth anon
snail ot request-join -p $PROJECT_DOMAIN -r
# accept request to join

# suppose you have a bunch of files
# /tmp/work/my-dir/a.tar.xz
# /tmp/work/my-dir/b.tar.xz
# /tmp/work/my-dir/subdirs-are-fine-too/c.tar.gz
# and you want to back up my-dir

cd /tmp/work
~/asset-sync-up.sh my-dir

And later:

cd /tmp/somewhere
~/asset-sync-down.sh my-dir

Technical info

Limitations:

  • Not meant for syncing from multiple places. Notably it doesn’t check the assets CDN for new stuff. Rather, it maintains its own metadata of what it last uploaded.
  • Normal files only. Directory structure is only recreated with mkdir- p.
  • File metadata is not preserved, e.g. permission bits, owner/group.
  • Uploads and downloads are all done with no parallelization lol!
  • It has no way to understand if you want to delete a file.
  • It doesn’t maintain your .glitch-assets file.
  • Files are uploaded as application/octet-stream type, so they don’t actually work well as assets. It really only is for backing up files.

Asset shadow area: These tools work with a directory, /app/.data/asset-sync-shadow, where it puts little “shadow” files corresponding to what files it has uploaded to the assets CDN. These shadow files are empty, and they have their modification time set to the modification time of the file last uploaded. The scripts also use the modification time on the shadow file to fill in that ?timestamp suffix on asset URLs, so that you don’t incorrectly download an outdated cached version of the asset.

Syncing “up” searches for files newer than their corresponding shadow file, and it uploads those.

Syncing “down” searches for shadow files newer than their corresponding file, and it downloads those.

This uses Snail :grinning: https://snail-cli.glitch.me/

4 Likes