How to create simple database or how to fix my .json database?

How I would be able to create simple database that can hold a lot of data and provide it in real time without visible delays?

So what I mean by this is that it’s simple as this:

var DatabaseManager = require("database-manager.js");
var login = (username, password) => {
  DatabaseManager.read("accounts").then((accounts) => {
    var account = accounts.filter(acc => acc.username == username);
    if (!account) return;
    if (account.password == password) {
      // login user
    } else {
      // oh no, password is incorrect!
    }
  })
}
var register = (username, password, email, avatar) => {
  // TODO: check if user doesn't exist already (we will ignore this step for now...)
  DatabaseManager.read("accounts").then((accounts) => {
    accounts.push({u: username, p: password, e: email, a: avatar}); // we won't hash the password as this is just example
    DatabaseManager.write("accounts", accounts);
  })
}

I tried similar thing but with .json database, but when I add avatar base64 url it breaks the database causing an json reading error. I tried using json asynced functions instead of normal ones, it fixed the issue but somehow after a while the errors keep appearing again.

  at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:61:3)
(node:13772) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 42)
(node:13772) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected end of JSON input

So, should I show the actual .json database thing that I made or should I replace it with some other database? Keep in mind, this is just small private projecting that I am working on, so I don’t care about securing anything at all.

The only thing that I want to store is my one array that holds all the accounts. Each account is made of:

{
  username: "Javascript",
  password: "myunsecurepassword1234",
  avatar: "data:image/png;base64,...",
  email: "javascript@official.com"
}

I am bit new to databases

You could try using the fs module - the file system is pretty fast and responds almost instantaneously. Keep in mind that I only read 10 characters from a plain text file though…

Yes I am using fs with .json file. However I am getting errors like I said.

If you want to fix up what you already have, I’d see if it helps to add some coordination to prevent simultaneous writes and reads. That might be causing the corrupted reads that lead to a syntax error when parsing the read JSON.

What do you mean by coordination, could you explain? Like not allowing to write and read at the same time? My database get’s overwritten every 1s. Should I try to cache the data instead of trying to read it every time? Write in database -> Update cache.

I’m going to paste a dictionary definition of “coordination” here:

the organization of the different elements of a complex body or activity so as to enable them to work together effectively.

which I think you understand. I go on to describe the idea more precisely as “to prevent simultaneous writes and reads,” which you also understand.

Right, referring to that, and also referring to not allowing more than one write at the same time.

It sounds like it’s already done so that it doesn’t normally do multiple writes at the same time. Do think about what if a single write takes more than one second though. But mainly what remains is to make sure you don’t try to read while it’s being written. I think filesystems don’t provide very nice guarantees about what you get from reading a file that’s currently being written. It could be a mix of old and new data or a prefix of the new data or something weirder.

1 Like
var express = require("express");
var app = express();
var server = require("http").createServer(app);
var io = require("socket.io")(server);
var port = process.env.PORT || 3000;
const fs = require("fs");

server.listen(port, () => console.log("Server listening at port %d", port));

app.use(express.static("public"));

var users = [
    {
        id: 0,
        username: "JavaScript",
        password: "this-is-not-my-password",
        avatar: "www.site.com/avatar.png",
        state: "on"
    }
];

function getRandomIntInclusive(min, max) {
    min = Math.ceil(min);
    max = Math.floor(max);
    return Math.floor(Math.random() * (max - min + 1) + min);
}

var current = 0;
var typing = [];
var active = [];

var app = { buildID: 0 };

function getUser(socket) {
    for (var i = 0; i < active.length; i++) {
        if (active[i].socket.id == socket.id) {
            return active[i].user;
        }
    }
    return null;
}

function getUsers() {
    var users = [];
    for (var i = 0; i < active.length; i++) {
        users.push(active[i].user);
    }
    return users;
}

function activateUser(socket, user) {
    if (doesUserExists(user.id)) return;
    active.push({
        socket: socket,
        user: {
            username: user.username,
            avatar: user.avatar,
            state: user.state
        }
    });
}

function doesUserExists(id) {
    for (var i = 0; i < active.length; i++) {
        if (active[i].user.id == id) { return true; }
    }
    return false;
}

var asyncStringify = (value) => { return new Promise((resolve, reject) => resolve(JSON.stringify(value))); }
var asyncParse = (value) => { return new Promise((resolve, reject) => resolve(JSON.parse(value))); }

var asyncWrite = (value) => {
    return new Promise((resolve, reject) => {
        asyncStringify(value).then((newValue) => {
            fs.writeFile("database.json", newValue, "utf8", () => resolve());
        });
    });
}

var asyncRead = () => {
    return new Promise((resolve, reject) => {
        fs.readFile("database.json", "utf8", (errors, contents) => {
            asyncParse(contents).then((newValue) => {
                resolve(newValue);
            });
        });
    });
}

asyncWrite(users);

io.on("connection", (socket) => {
    current++;
    io.sockets.emit("status-viewers", current);
    socket.on("server-message", (data) => {
        if (data.content.length >= 2000) { socket.emit("alert", "Too much work!"); return; }
        data.id = getRandomIntInclusive(1, 100000000000);
        io.sockets.emit("client-message", data);
    });
    socket.on("server-message-delete", (data) => {
        io.sockets.emit("client-message-delete", data);
    });
    socket.on("server-message-edit", (data) => {
        io.sockets.emit("client-message-edit", data);
    });
    socket.on("server-login", (data) => {
        for (var i = 0; i < users.length; i++) {
            if (users[i].id == data.id) {
                socket.emit("client-login", users[i]);
                activateUser(socket, users[i]);
                io.sockets.emit("status-active", getUsers());
                return;
            }
        }
        socket.emit("client-login", null);
    });
    socket.on("server-typing", (data) => {
        try {
            if (!data) return;
            if (data.typing && data.user) {
                var duplicated = false;
                for (var i = 0; i < typing.length; i++) {
                    if (
                        typing[i].hasOwnProperty("id") &&
                        data.user.hasOwnProperty("id")
                    ) {
                        if (typing[i].id == data.user.id) {
                            duplicated = true;
                        }
                    }
                }
                if (!duplicated) {
                    typing.push(data.user);
                }
            } else {
                for (var i = 0; i < typing.length; i++) {
                    if (typing[i] && data.user) {
                        if (typing[i].id == data.user.id) {
                            typing.splice(i, 1);
                            break;
                        }
                    }
                }
            }
            io.sockets.emit("client-typing", typing);
        } catch (e) { console.log(e); }
    });
    socket.on("update-state", (data) => {
        try {
            if (!data.user) return;
            fs.readFile("database.json", "utf8", (errors, contents) => {
                asyncParse(contents).then((accounts) => {
                    for (var i = 0; i < accounts.length; i++) {
                        if (accounts[i].id == data.user.id) {
                            accounts[i].state = data.user.state;
                            asyncStringify(accounts).then((value) => {
                                fs.writeFile("database.json", value, "utf8", () => io.sockets.emit("status-active", accounts));
                            });
                            break;
                        }
                    }
                });
            });
        } catch (e) { console.log(e); }
    });
    socket.on("account-change-avatar", (data) => {
        try {
            fs.readFile("database.json", "utf8", (errors, contents) => {
                asyncParse(contents).then((accounts) => {
                    for (var i = 0; i < accounts.length; i++) {
                        if (accounts[i].id == data.id) {
                            accounts[i].avatar = data.url;
                            asyncStringify(accounts).then((value) => {
                                fs.writeFile("database.json", value, "utf8", () => socket.emit("account-refresh", data.url));
                            });
                        }
                    }
                });
            });
        } catch (e) { console.log(e); }
    });
    socket.on("disconnect", () => {
        current--;
        io.sockets.emit("status-viewers", current);
        for (var i = 0; i < active.length; i++) {
            if (active[i].socket.id == socket.id) {
                active.splice(i, 1);
                break;
            }
        }
        io.sockets.emit("status-active", getUsers());
    });
});

setInterval(() => { io.sockets.emit("update-check", app); }, 1000);

The code is bit of a mess, because I kept trying to change the code and implement new ways to save and read database without breaking it.

Do you see where or what may be causing the error?

and yes I used this not recommended method to quickly suppress the errors:

this is why i like .then and .catch lol

1 Like

Yeah I know lol,
Do you guys see why my code keeps throwing errors related to json unexcepted end thing?

The asyncParse function has

JSON.parse(value)); 

So its being called with a string that is either empty or is not a full JSON, i.e. truncated.

You’re basically implementing your own database, you’ll need to deal with similar issues that they’re written for, i.e. caching and file locking.

If you want to use a database implementation written already, there are heaps of alternatives, some examples are listed in https://glitch.com/@Closer2U/databases

1 Like

For json dbs, I’d recommend jsoning made by @khalby786. It’s really easy to use and I’ve never had any problems with it.

3 Likes

Yeah I am writing my own database. I think I know how to fix this.

  1. I will need to remove that constant writing and reading.
  2. I will only read and write into cache.
  3. Cache will be saved into database every minute. If it’s corrupted it will not be saved and it will throw an warning to warn me that it’s unable to save. It will save the data to plain txt file backup.txt, instead. It won’t start saving again if it’s still writing.
  4. Improve overall system.
  5. I will only read database on startup, if it’s corrupted it will use default cache.
1 Like