Why is fs.readFileSync() faster than await fsPromises.readFile()?

Question

Here is the test code (in an express environment just because that's what I happen to be messing around with):

const fs = require('fs-extra');
const fsPromises = fs.promises;
const express = require('express');
const app = express();

const speedtest = async function (req, res, next) {
    const useFsPromises = (req.params.promises == 'true');
    const jsonFileName = './json/big-file.json';
    const hrstart = process.hrtime();
    if (useFsPromises) {
        await fsPromises.readFile(jsonFileName);
    } else {
        fs.readFileSync(jsonFileName);
    }
    res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
};

app.get('/speedtest/:promises', speedtest);

The big-file.json file is around 16 MB. Using node 12.18.4.

Typical results (varies quite a bit around these values, but the following are "typical"):

https://dev.mydomain.com/speedtest/false
time taken to read: 3.948152 ms

https://dev.mydomain.com/speedtest/true
time taken to read: 61.865763 ms

UPDATE to include two more variants... plain fs.readFile() and also a promisified version of this:

const fs = require('fs-extra');
const fsPromises = fs.promises;
const util = require('util');
const readFile = util.promisify(fs.readFile);
const express = require('express');
const app = express();

const speedtest = async function (req, res, next) {
    const type = req.params.type;
    const jsonFileName = './json/big-file.json';
    const hrstart = process.hrtime();
    if (type == 'readFileFsPromises') {
        await fsPromises.readFile(jsonFileName);
    } else if (type == 'readFileSync') {
        fs.readFileSync(jsonFileName);
    } else if (type == 'readFileAsync') {
        return fs.readFile(jsonFileName, function (err, jsondata) {
            res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
        });
    } else if (type == 'readFilePromisified') {
        await readFile(jsonFileName);
    }
    res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
};

app.get('/speedtest/:type', speedtest);

I am finding that the fsPromises.readFile() is the slowest, while the others are much faster and all roughly the same in terms of reading time. I should add that in a different example (which I can't fully verify so I'm not sure what was going on) the time difference was vastly bigger than reported here. Seems to me at present that fsPromises.readFile() should simply be avoided because there are other async/promise options.

`fs.readFileSync()` is a completely different implementation and I suspect that it can just go faster because it can do things differently when it is allowed to entirely hog the CPU and never has to go back to the event loop while reading the file, whereas `fs.readFile()` reads in chunks, going back to the event loop for each chunk. Each of these trips through the event loop likely costs some time. In general, you pick `fs.readFile()` when you don't want to block the event loop so not blocking the event loop is more important than overall execution time. — jfriend00, Sep 19 '20 at 19:55
It's also possible (I did not take the time to track this down in the node.js code) that `fs.readFile()` reads in smaller chunks at a time to make it's time blocking the event loop into shorter intervals. For a different project, I wrote my own version of `fs.readFile()` that read in very large chunks and it was meaningfully faster (I was not concerned about blocking the event loop in that project). — jfriend00, Sep 19 '20 at 20:02
That wrong. readFileSync takes more time and more CPU usage. The readFile is using another event loop, new thread. learn more about it. — Talg123, Sep 20 '20 at 05:14

jfriend00 · Answer 1 · 2020-09-21T02:13:28.490

After stepping through each implementation in the debugger (fs.readFileSync and fs.promises.readFile), I can confirm that the synchronous version reads the entire file in one large chunk (the size of the file). Whereas fs.promises.readFile() reads 16,384 bytes at a time in a loop, with an await on each read. This is going to make fs.promises.readFile() go back to the event loop multiple times before it can read the entire file. Besides giving other things a chance to run, it's extra overhead to go back to the event loop every cycle through a for loop. There's also memory management overhead because fs.promises.readFile() allocates a series of Buffer objects and then combines them all at the end, whereas fs.readFileSync() allocates one large Buffer object at the beginning and just reads the entire file into that one Buffer.

So, the synchronous version, which is allowed to hog the entire CPU, is just faster from a pure time to completion point of view (it's significantly less efficient from a CPU cycles used point of view in a multi-user server because it blocks the event loop from doing anything else during the read). The asynchronous version is reading in smaller chunks, probably to avoid blocking the event loop too much so other things can effectively interleave and run while fs.promises.readFile() is doing its thing.

For a project I worked on awhile ago, I wrote my own simple asynchronous version of readFile() that reads the entire file at once and it was significantly faster than the built-in implementation. I was not concerned about event loop blockage in that particular project so I did not investigate if that's an issue.

In addition, fs.readFile() reads the file in 524,288 byte chunks (much larger chunks that fs.promises.readFile()) and does not use await, using just plain callbacks. It is apparently just coded more optimally than the promise implementation. I don't know why they rewrote the function in a slower way for the fs.promises.readFile() implementation. For now, it appears that wrapping fs.readFile() with a promise would be faster.

What about `fs.readFile()`, i.e. the plain async version? I am finding that to have roughly the same read time as `fs.readFileSync()` and yet being asynchronous it should also suffer from the same drawbacks that `fs.promises.readFile()` does? — drmrbrewer, Sep 20 '20 at 08:19
I've updated my question with `fs.readFile()` and also a promisified version of `fs.readFile()` added. — drmrbrewer, Sep 20 '20 at 08:41
@drmrbrewer - Well, `fs.readFile()` reads the file in 524,288 byte chunks and does not use `await`. It is apparently just coded more optimally than the promise implementation. I don't know why they rewrote the function in a slower way for the `fs.promises.readFile()` implementation. For now, it appears that wrapping `fs.readFile()` with a promise would be faster. — jfriend00, Sep 21 '20 at 02:11

Why is fs.readFileSync() faster than await fsPromises.readFile()?

1 Answers1

Linked