Glitch, could you tell us the technical findings from the outage investigation so far?

[wh0]

I’m posting this to encourage the team to feel free to post their technical findings as they’re investigating this ongoing outage. I get the feeling from the latest update from status.glitch.com that they’re focusing on really… bottom-line is-it-working-yet kind of updates:

Hey there! No real news, but we wanted to let you know that we are still here working to stabilize the system, and we thank you for checking in with us. We know this has been a long one

I don’t mind at all reading technical updates even if otherwise you consider it “no real news.” And I think there are many people on here to learn stuff, and we’d be interested in knowing what’s going on.

[tasha]

I totally agree @wh0! The issue with these last few incidents is that there is little new information to provide.

As a result, the updates end up being used more to reassure the community that we hear them, we are really sorry, and we are still working hard to get everything back online.

I know that’s really frustrating! These type of updates are really frustrating to write too!

Here’s what we can do…
Once the incident is resolved, our team will have a retrospect on what happened. I’ll see what information from the retro I can share with the community. I’ll post about it in the incident thread here: Service Incident Update: Projects Loading Slowly or Not At All, NPM Reinstalls, & Missing Code

[SebPautot]

I’ll probably dive into that, I’m interested in learning how Glitch works/does not work.

[wh0]

Thanks for the reply. I just saw that you posted this small update:

[Resolved] Service Incident Update: Projects Loading Slowly or Not At All, NPM Reinstalls, & Missing Code

Hello! I just got out of a meeting about the incident and wanted to share some additional info with you about what is going on: We had launched a few changes last week that significantly altered the way projects run. This caused some unexpected side effects that we have been working on since. We are making some massive changes to our infrastructure to ultimately get to a place of much higher stability for the Glitch platform. Because these changes are large and affect critical pieces of our i…

Too bad that the team wasn’t able to share any technical findings yet. I really appreciate that you must have fought hard in that meeting to get at least this statement agreed upon.

What are the circumstances for talking about this? Is it too risky to mention anything until the team is absolutely certain, for fear of harming business relationships with providers?

[javaarchive]

Overall I’d like to say the way glitch shares packages is really smart! Services like repl.it spend 90% of their time when you click run on installing packages. This is time-consuming and also makes the lifetime of the disks shorter.

[wh0]

@tasha

Here’s what we can do…
Once the incident is resolved, our team will have a retrospect on what happened. I’ll see what information from the retro I can share with the community. I’ll post about it in the incident thread here: Service Incident Update: Projects Loading Slowly or Not At All, NPM Reinstalls, & Missing Code

I saw that the incident was marked as resolved recently. This retrospect could be right around the corner!