Large JSON file problem (static database)

Glitch Help
,
#1

Hola :monkey:

I have an issue with a 261mb json file that gets corrupted while uploading :confused:
I tried uploading through the vscode extension (no error, but filesize 0b), the webbrowser (crashes) and ultimately imported through github (using LFS).

I am importing the json file with const ffhq = require('./data/ffhq-dataset-v2.json') in my server.js and locally that worked fine. If I upload a smaller size json it is also working.

I even upgraded my account because I thought that the 200mb limitation would be the problem, but after boosting my app I am still only using 3mb/400mb which is weird…

Here is the link if somebody wants to look at the code: https://glitch.com/edit/#!/this-person-does-exist
The error I am getting is:

internal/modules/cjs/loader.js:787
    throw err;
    ^
SyntaxError: /app/data/ffhq-dataset-v2.json: Unexpected token v in JSON at position 0
    at JSON.parse (<anonymous>)
    at Object.Module._extensions..json (internal/modules/cjs/loader.js:784:27)
    at Module.load (internal/modules/cjs/loader.js:628:32)
    at Function.Module._load (internal/modules/cjs/loader.js:555:12)
    at Module.require (internal/modules/cjs/loader.js:666:19)
    at require (internal/modules/cjs/helpers.js:16:16)
at Object.<anonymous> (/app/server.js:4:14)
    at Module._compile (internal/modules/cjs/loader.js:759:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:770:10)
    at Module.load (internal/modules/cjs/loader.js:628:32)
#2

That is way too big for glitch. Even boosted, you only have 200 mb.

#3

It says 400mb though…
And why do I not get an error message that the file is to big, instead it seems to just corrupt the file :dizzy_face:

How could I fix this? Is it possible to convert it into another kind of database?

chrome_jGcAB3aktC
chrome_jGcAB3aktC2525×551 43.1 KB

#4

JSON files are prone to corruption, that isn’t Glitch’s fault.

Maybe try something like this?

1 Like
#5

You’re mistaken. Our community had talked about using raw uncoordinated raw filesystem operations to manage application state on disk–many examples of which incidentally serialized data in JSON–as being more prone to corruption compared to a database management system.

Transferring a file without corruption is something Glitch should be able to do. But on the other hand, I think Glitch will soon stop supporting their VS Code extension, so you might have to transfer it some other way.

1 Like
#6

Thank you for all the replies.
It really seems that something happens to the file in transition.
As I said before Chrome crashes, VSCode only uploads ~1kb and using git lfs I only get the metadata :frowning:

Any other way to upload a file?

Otherwise I’ll try to convert it into another database.

#7

Ok, I was able to use wget to get the file from my dropbox :slight_smile:
This seemed to be an unnecessary pain point.
Now my app is using 350/400mb space and 1/2gb RAM … living on the edge, haha.

But it’s working!

1 Like
#8

Good to hear!

1 Like