NodeJS Running through file by steams

Question

I'm trying to read through a file using a stream. The file is a custom file format that has some number of sections, each being 66 bytes. I have to check the value of the the first 32 bytes against a given 32 bytes, and if they match return the remaining 34 bytes. Currently I'm doing the following

const fs = require('fs')

/**
 * @param {string} known
 */
function check(known) {
  let stream = fs.createReadStream('path/to/file', {highWaterMark: 66});
  let out = '';
  stream.on('data', (chunk) => {
    let temp = '';
    for (let i = 0; i < 32; i++) {
      temp += String.fromCharCode(chunk.at(i));
    }
    if (temp === known) {
      for (let i = 32; i < 66; i++) {
        out += String.fromCharCode(chunk.at(i));
      }
    }
  });
  return out;
}

However (from checking with console.log) I know that the function in stream.on is run after check "finishes" so regardless of if the given string is found, the returned value will always be the same. How fix? Thanks in advance

jfriend00 · Accepted Answer · 2022-03-09T00:04:43.487

Streams are non-blocking and asynchronous. They call their event handlers some time in the future, after your check() function has returned. That is how they work.

As such, you cannot directly return your result from the function because the function returns before the result is known. Instead, you have to return the result using an asynchronous mechanism such as an event, a callback or a promise (I would prefer a promise in this case). see How to return the response from an asynchronous call for more detail on returning your asynchronously retrieved value.

But, using a stream to read a file that is structured in 66 byte chunks is the hard way to write this code because data blocks are going to be randomly sized and are not guaranteed to line up with your 66 byte blocks. It would be simpler to use const fileHandle = await fs.promises.open(...) and fileHandle.read(...) to read a 66 byte chunk from the beginning of the file, check the 32 bytes you want to check and then communicate back the remaining 34 bytes using a promise.

Here's one such way to write it using handle.read() instead:

const fsp = require('fs').promises;

async function check(known) {
    const kReadLen = 66;
    const kHeaderLen = 32;
    let handle = await fsp.open('path/to/file');
    try {
        let buffer = Buffer.alloc(kReadLen, 0);
        let { bytesRead } = await handle.read(buffer, 0, kReadLen, 0);
        if (bytesRead !== readLen) {
            throw new Error("Insufficient data in the file");
        }
        const headerStr = buffer.toString('utf8', 0, kHeaderLen);
        if (headerStr === known) {
            return buffer.toString('utf8', kHeaderLen, kReadLen);
        } else {
            return "";
        }
    } finally {
        handle.close();
    }
}

Sample Usage:

check(testKnown).then(result => {
    console.log(result);
}).catch(err => {
    console.log(err);
})

NodeJS Running through file by steams

1 Answers1