My JSON files keeps breaking when writing for some reason

Glitch Help
1

So, after it breaks, the JSON file looks like this

{
  "server": {
    "bans": {},
    "mutes": {},
    "kicks": {},
    "warns": {},
    "configs": {
      "punishLogs": "false",
      "punishLogchannel": "",
      "actionLogs": "false",
      "actionsLogsChannel": "",
      "appLogs": "false",
      "appLogsChannel": "",
      "xprate": "1",
      "levelRewards": {},
      "levelMsgs": "false",
      "levelChannel": "",
      "levelMsg": "Congrats {member}! You've just hit level {level}!",
      "premium": "true"
    },
    "levels": {
      "266162824529707008": {
        "level": 1,
        "xp": 52
      }
    }
  }
}
      }
    }
  }
}

The code I use to write is like this

const Discord = require('discord.js');
const fse = require('fs-extra');

module.exports = {
    name: "config",
    description: "Configure the bot for your server.",
    alaises: ['configs', 'settings', 'guildsettings', 'dashboard'],
    usage: '<setting> <arguments>',
    category: 'manager',
    guildOnly: true,
    example: 'No example.',

    code(client, message, args) {
        if (!message.member.hasPermission('ADMINISTRATOR')) return message.reply(`You are not permitted to configurate the bot for this server as you are not an administrator.`);
        fse.readJson(`./databases/${message.guild.id}.json`, (err, db) => {
            if (err) return message.reply(`Error, ${err}`).then(console.error(err));
//A lot of code.
db.server.configs.punishLogs = 'true';
return fse.writeFile(`./databases/${message.guild.id}.json`, JSON.stringify(db, null, 2))

Any way to make it not break my JSON files?

2

When it breaks, what does the error look like?

3

There is no error before I try to use the command again, and it gives @Apex, Error, SyntaxError: ./databases/623191963000242176.json: Unexpected end of JSON input

4

It seems to only happen to this one server as well.

image
image1291Ă—878 101 KB

5

You want to make sure that while the file is being read, it is not at the same time being written to, if so its likely to read empty. And not being written to while at the same time being written to by another change. And not being written to after being written to but since the corresponding read.

This is called locking, and is part of the process of writing your own database code. Instead, you could use a database package where it has all been done for you.

2 Likes
6

I honestly prefer using JSON.
Here is my full config.js file
Perhaps you will be able to find out where the issue is in there.

7

Maybe it could be made robust without locking if using synchronous file reads and writes, and making sure no asynchronous code comes between the read and the write.

The disadvantage would be, the bot would appear slower, as it would no longer be processing commands while waiting for a file read or write to be completed.

8

As you may have seen, what I did is that I read the file, then I put in all the if statements to be sure as for what to put, and I use the same readJson method in fs-extra for the entire command. Though, for each command, only one object will be modified.

9

Oh wait, if I understand this right here.
You mean putting the read and write together when I am writing?

10

Thanks for sharing the code, it helps understand.

fse.writeFile is an asynchronous function, that is, while the file system sends data to the file, Node is processing more of your code, which could be another command if they come close together in time.

There are synchronous file read and write functions available, I’m not sure if in fse, but definitely in fs.

11

fse is exactly the same as fs, just with some extra features, such as readJson.
Well, so, should I make it into fse.writeFileSync()?

12

Yes, but more than that, the entire code function needs to be synchronous! :slight_smile:

Which would be a problem, because these parts are also asynchronous …

message.channel.send
message.author.send
message.guild.channels.find
13

Oh god, I always use those. I actually don’t have any other ways of doing it.

14

I just checked, and fse.readJson is asynchronous as well.

15

You could keep the code async as it is, and implement file locking.

Plus reading and writing the entire file at a time is unnecessary overhead, so you’d want to cache in memory, but decide when to store to disk.

Its fun but its reinventing the wheel, lol!

Check out the code samples in another database package, quickdb, which is a wrapper on top of SQLite …

https://quickdb.js.org/

16

Neither I or my development partner can work SQLite, we just know how to do JSON.

17

But ok so, the file locking, how would I do that?

18

From quickdb docs …

Quick.db is an easy to use database wrapper for better-sqlite3, it was designed to be simple to let new users who are just getting into development and don’t want to worry about learning SQL just quite yet.

// Setting an object in the database:
db.set('userInfo', { difficulty: 'Easy' })

(I’d prefer nedb myself but this shows why quickdb is so popular)

Because you are rewriting the entire file each time, you can’t let another command even look at the file while one command is processing, so …

start of the function, before reading the file:

  • check if a lock file exists
  • check if lock file is “old” and should be expired (somehow the previous command didn’t complete)
  • build those checks into async code that effectively blocks the command until the lock is cleared

at the end of the function, after writing the file:

  • delete the lock file

it might also need to handle process.exit (ie. the project is being restarted or shutdown)

19

May I get a example code of how that would be? It’s like 2 am right now, I’ve been working on this since 10 am yesterday. I am really tired, so it’s hard for me to think about examples in my head right now.

20

Although its fun, its also hard to get it right. I won’t write this code unless its a paid gig, and I doubt you would pay someone to rewrite a cut-down database engine …

Note that writing that code to work well is hard, and has been done already by every database engine.

Check out the complexity involved in even a simple JSON database: nedb/lib at master · louischatriot/nedb · GitHub