How to be sure two clients are not requesting at the same time corrupting state

Question

I am new to Node JS and I am building a small app that relies on filesystem.

Let's say that the goal of my app is to fill a file like this:

1
2
3
4
..

And I want that at each request, a new line is written to the file, and in the right order.

Can I achieve that?

I know I can't let my question here without making any code so here it is. I am using an Express JS server:

(We imagine that the file contains only 1 at the first code launch)

import express from 'express'
import fs from 'fs'

let app = express();

app.all('*', function (req, res, next) {
    // At every request, I want to write my file
    writeFile()
    next()
})

app.get('/', function(req,res) {
    res.send('Hello World')
})

app.listen(3000, function (req,res) {
    console.log('listening on port 3000')
})

function writeFile() {
    // I get the file
    let content = fs.readFileSync('myfile.txt','utf-8')

    // I get an array of the numbers
    let numbers = content.split('\n').map(item => {
        return parseInt(item)
    })

    // I compute the new number and push it to the list
    let new_number = numbers[numbers.length - 1] + 1
    numbers.push(new_number)

    // I write back the file
    fs.writeFileSync('myfile.txt',numbers.join('\n'))
}

I tried to make a guess on the synchronous process that made me thinking that I was sure that nothing else was made at the same moment but I was really note sure...

If I am unclear, please tell me in the comments

Answer 1

If I understood you correctly, what you are scared of is a race condition happening in this case, where if two clients reach the HTTP server at the same time, the file is saved with the same contents where the number is only incremented once instead of twice.

The simple fix for it is to make sure the shared resource is only access or modified once at a time. In this case, using synchronous methods fix your problem. As when they are executing the whole node process is blocked and will not do anything.

If you change the synchronous methods with their asynchronous counter-parts without any other concurrency control measures then your code is definitely vulnerable to race conditions or corrupted state.

Now if this is only the thing your application is doing, it's probably best to keep this way as it's very simple, but let's say you want to add other functionality to it, in that case you probably want to avoid any synchronous methods as it blocks the process and won't let you have any concurrency.

A simple way to add a concurrency control, is to have a counter which keeps track how many requests are queued. If there's nothing queued up(counter === 0), then we just do read and write the file, else we add to the counter. Once writing to the file is finished we decrease from the counter and repeat:

app.all('*', function (req, res, next) {
  // At every request, I want to write my file
  writeFile();
  next();
});

let counter = 0;

function writeFile() {
  if (counter === 0) {
    work(function onWriteFileDone() {
      counter--;
      if (counter > 0) {
        work();
      }
    });
  } else {
    counter++;
  }

  function work(callback) {
    // I get the file
    let content = fs.readFile('myfile.txt','utf-8', function (err, content) { 
      // ignore the error because life is too short on stackoverflow questions...

      // I get an array of the numbers
      let numbers = content.split('\n').map(parseInt);

      // I compute the new number and push it to the list
      let new_number = numbers[numbers.length - 1] + 1;
      numbers.push(new_number);

      // I write back the file
      fs.writeFile('myfile.txt',numbers.join('\n'), callback);
    });
  }
}

Of course this function doesn't have any arguments, but if you want to add to it, you have to use a queue instead of the counter where you store the arguments in the queue.

Now don't write your own concurrency mechanisms. There's a lot of in the node ecosystem. For example you can use the async module, which provide a queue.

Note that if you only have one process at a time, then you don't have to worry about multiple threads since In node.js, in one process there's only one thread of execution at a time, but let's say there's multiple processes writing to the file then that might make things more complicated, but let's keep that for another question if not already covered. Operating systems provides a few different ways to handle this, also you could use your own lock files or a dedicated process to write to the file or a message queue process.