"Glitchy" misrepresentation that "Too many requests.." are being made


#1

Glitch recently implemented a throttling system which falsely reports “Too many requests…” are being made. This often happens immediately after remixing a project even on the first run with one user connected. :confused:


#2

Hi Eric, sorry to hear that - can you provide us with the name of the project that you’re having trouble with so we can dig into this. Thanks.


#3

Hello Gareth,
Many of the projects which had worked fine until a little over a week ago are now experiencing that issue. I notice it with many of my students working with remixes of projects such as:



and

Best,
Eric


#4

An important note is that this has been happening even on the first http request immediately after remixing the project with the show live giving a full green light. The index.html isn’t even sent in these cases.


#5

Hi @EricEisaman thanks for additional details. We’ll certainly take a look to see if there’s something about how these projects start up that is particularly problematic or is incorrectly triggering rate-limiting. Given the timeline you mention the change that’s probably affecting these projects is work that we did to better enforce the existing stated restrictions from the technical restrictions, so it’s possible that these projects are simply being rate-limited properly now where they weren’t before.

We’ll test some remixes and let you know what we find; thanks for the report!


#6

I am positive that many such “Too many requests…” responses were literally to the first http requests ever sent to the projects. I conducted some remix testing which showed exactly that during our classes this past week.


#8

Howdy @SirFizX I’ve played with these projects and remixed them a few times and have yet to see any rate-limited behavior. I also scripted out requests to a remix of the last of your example projects and got to around a hundred full-page requests before I started to see rate-limiting rejections. For the record, here’s what I’m doing in my script:

i=0;while [ $i -lt 8000 ]; do; wget --page-requisites http://saber-mall.glitch.me/ 2>&1 | grep 'ERROR 429' ; i=$[$i+1]; echo $i; sleep .5; done;

Can you provide any additional details? Are more than one person connecting to the newly-remixed projects when you’re seeing this behavior? Do you happen to have a capture that displays the issue that we can take a closer look at?


#9

Good day @cori. Thank you for looking into this and providing your test script. My students encounter this problem primarily at school where all our http requests go through about 10 extra hops due to the network architecture and state network security design. In many cases I believe the request count mounts very quickly due to a failure in establishing a wss with the projects backends. The current default is the socket.io standard default of falling back to http polling which quickly runs up the requests count. Our district subscribes to Lightspeed Systems web filter service. Our network engineer recently upped his game in understanding their method for setting general whitelisting rules for wss connections. I will reach out to him to ensure such a general whitelisting rule for wss connections on your domain is in place.

All that said, there have been times where remixes returned ‘ERROR 429’ in the first go. I will run a similar script on students computers once our classes resume.

Thank you so much for your effort. I will surely acquire documentation and get back to you in two weeks. In the meantime I wish you and your family a wonderful holiday season!

Cheers,
Eric


#10

Thanks a lot for the response! I look forward to hearing more in a few weeks, and hope you enjoy your holiday!


#11

Hello @cori . We determined that Lightspeed was blocking the SSL cert during the http upgrade request for a secure websocket. We were able to create an SSL exception in our Lightspeed account. The problem no longer exists. Thank you for your help.
Sincerely,
Eric


#12

Hey @SirFizX, I’m glad to hear you found the problem! Thanks for letting us know.

Happy Glitching!