"The app exceeded the memory limit." even though it didn't, according to the container stats

Glitch Help
1

I’m having some problems with the memory limiter and/or the accuracy of the “container stats”. Here’s a minimal example I created:

https://glitch.com/edit/#!/exceeded-memory-limit

And here’s what always happens after about 10 minutes of downloading and decompressing the compressed archive file:

I’m using this function (from here) to get the memory usage:

function getMemoryUsage() {
  let total_rss = Number( require('fs').readFileSync("/sys/fs/cgroup/memory/memory.stat", "utf8").split("\n").filter(l => l.startsWith("total_rss"))[0].split(" ")[1] ) / 1e6; 
  return Math.round( total_rss - (process.memoryUsage().rss/1e6) );
}

Any ideas what’s going on here?

Cheers,
Joe

3

Thanks for reporting this. I’m sure of how the stats UI and logs differ in where they’re getting their information from so will run this by the dev team and will find out.

4

Hi @joe :slight_smile:

Took me a while to understand where the problem was! In getMemoryUsage, you’re subtracting the memory usage of your process, so the thing you’re measuring is how much memory our internal server is using :slight_smile:

Just return total_rss, or total_rss minus a fixed amount of memory (say 60MB, it won’t take much more than that anyway). Hope this helps :slight_smile:

5

Thanks! @cori, it might be helpful to update your post here to reflect what @etamponi said.

I’m now seeing a sudden “jump” from ~190mb to 500mb just before the “exceeded” message:

And watching htop at the same time shows that it jumps from this:

image
image.png1118×336 58.3 KB

To this (just before the “exceeded” message):

image
image.png1119×341 60.5 KB

Could very well be something wrong with the module, but it seems very strange and I’m pretty sure I had the same problem with the lzma-native module. I’ll keep investigating.

Update: Looks like it is a problem with the node-zstd module. The lzma-native module processes a similar file well past the 10k lines mark (it’s at 30k as of writing with no sign of stopping):

https://glitch.com/edit/#!/exceeded-memory-limit-lzma-native