[Resolved] Service Incident Update: Projects Loading Slowly or Not At All, NPM Reinstalls, & Missing Code

Ongoing Incident Update - We are still working to bring the site fully back up.

Once this incident is marked “Resolved” on status.glitch.com, most of these issues will fix themselves.

  • loading slowly or not at all
  • not responding
  • github import/export errors
  • npm reinstalls

Folks with the following issues will need to send a message to support@glitch.com so we can help you out:

  • suspension for preparing error
  • suspension for disk space
  • missing code - if you are missing code, don’t open the project editor further so we can restore your project to a backup.
4 Likes

Yesterday, I had put multiple hours of time into one of my glitch projects, to log in today and find all of it gone, completely unsaved despite it being saved properly yesterday. I checked rewind to see if something happened, and it appears that everything that happened yesterday is no longer in rewind except a single change that happened at the start of the day.

Is there anything I can do to recover the changes made yesterday? and what happened to all of the stuff that was saved properly yesterday.

1 Like

I’ve had this issue as well! I noticed that it happened after I remixed the project, which was helpful because the remix served as a makeshift backup, though it was tragic for me to experience and I’m now paranoid that none of my progress will actually be saved as well.

It’s very severe, it will probably take hours to reimplement everything that is lost. We did remixes of the project earlier but none of them yesterday, and there seems to be no backups or anything. I’ve seen glitch having a lot of connection problems throughout the last few days but this is the worst thing that has happened yet.

Try contacting : support@glitch.com
Do no make any modification as it might erase previous version.

1 Like

I’ve already started rewriting a bunch of it, and I don’t really need to get it restored, but it’s still very upsetting that glitch did this, and my fear is that it may happen again. Going through support and not modifying the project at all will probably take longer then just rewriting the deleted code (and I’ve also already modified the project).

This happened to me a long time ago and I got it back within 24 hours, but this was weeks of work. It seemed to get reverted as if someone at Glitch Support reverted it back to a previous backup (e.g. Rewind was reverted as well) without me asking (they didn’t). Anyway they can get it back, but it seems this is a returning issue for Glitch users so maybe @glitch_support could look into it :slight_smile:

Thanks for letting us know!

When I ran enable-pnpm I got this error:
ERROR ENOSPC: no space left on device, open ‘/rbd/pnpm-volume/store/2/store.json.1783575922’
I checked and im no where near 400mb. Nothing works now since I keep getting a can’t find module error.

Edit: This is where it all begins… (cus it was moved lmao)

2 Likes

@glitch_support please help I have no idea why this is happening

Use git gc or git prune to get a little bit of a space.

1 Like

Already tried that and it did nothing

Try with rm -rf /app/.git/. It should works.

After running that and doing git gc this happened so it didn’t fix :frowning:

No space left :joy:

There is space. SPACE!!!

what… :no_mouth:

Good call posting it here. /rbd/pnpm-volume/store/2/store.json.1783575922 is in the bare container disk, i.e. not in the project’s /app “400 MB limit” drive, and not in the node_modules drive either, and not even on that /tmp mount from outside. It makes sense that clearing out the /app/.git dir, which lives in the project drive, didn’t fix it. Sorry about your project’s commit history, btw.

But just for the sake of having a written record, could you post the output of this:

df -h / /tmp /app

thanks

If there’s anything big in your container’s /tmp, maybe try erasing those \:
… but if this server is really under a lot of disk pressure, you might not even get that space back ):

Gimme a second I’m on mobile