0

I need to know the encoding of a node stream for which I am using detect-character-encoding module. But the problem is that I can only read encodings of a buffer and not a stream due to which I have to do something like this:

FileStream.on('data', (chunk) => {
  console.log(chunk)
  const charsetMatch = detectCharacterEncoding(chunk)
  console.log(charsetMatch)
})

Knowing stream encoding comes at the cost of losing a chunk of data, which is required later in the code flow. Is there a way possible in which I can just peek at chunk know its encoding and not lose the chunk and data?

Mayank Patel
  • 346
  • 3
  • 18

1 Answers1

1

You can build a promise to return both the contents and the charset of the stream:

const charsetStream = (stream) => new Promise((resolve, reject) => {

  const detectCharacterEncoding = require('detect-character-encoding');
  let chunks = [];

  stream.on('data', (chunk) => {
    chunks.push(chunk);
  })

  stream.on('end', () => {
    chunks = Buffer.concat(chunks);
    resolve({
      content: chunks,
      charset: detectCharacterEncoding(chunks)
    })
  })

  stream.on('error', (err) => {
    reject(err);
  })

});

charsetStream(FileStream)
  .then(info => {
    console.log('content', info.content);
    console.log('charset', info.charset);
  })
  .catch(console.log);
  
  // You can use the FileStream outside the method but you can use it once !
  // this is completely different than the "stream" variable
  FileStream.on('data', (chunk) => {
    console.log('FileStream', chunk.toString());
  })
darklightcode
  • 2,738
  • 1
  • 14
  • 17
  • In the above code snippet, is there a way wherein which upon detecting encoding I am able to regenerate the fileStream, from that array of chunks? because there are several functionalities in the code after this which expect a stream – Mayank Patel Sep 02 '21 at 08:42
  • No you can't, streams are meant to be consumed and that's it, however you can pipe it to two new streams and consume those, stream1 to detect encoding and stream2 to do-whatever. But this is something people don't do, you rather store the consumed stream in a variable that's why i kept the contents of the stream in the output. If you're mind is set on keeping the main stream check out this post [https://stackoverflow.com/questions/54348804/node-js-copy-a-stream-into-a-file-without-consuming](https://stackoverflow.com/questions/54348804/node-js-copy-a-stream-into-a-file-without-consuming) – darklightcode Sep 02 '21 at 08:51
  • I have updated my answer at the end of code, perhaps it will be more clear for you. – darklightcode Sep 02 '21 at 09:03