How to pass a Buffer as argument of fs.createReadStream

Question

According to docs https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options

fs.createReadStream() can accept Buffer as first argument

my node code:

var _ = require('lodash')
var faker = require('faker')
var http = require('http')
var fs = require('fs')
var xlsx = require('node-xlsx')

var gg = _.range(10).map((item) => {
  return _.range(10).map((item) => {
    return faker.name.findName()
  })
})

http.createServer(function(req, res) {
  var buf = xlsx.build([{
    name: 'sheet1',
    data: gg
  }])
  fs.createReadStream(buf, 'binary').pipe(res)

}).listen(9090)

but I get this error:

events.js:160
  throw er; // Unhandled 'error' event
  ^

Error: Path must be a string without null bytes
at nullCheck (fs.js:135:14)
at Object.fs.open (fs.js:627:8)
at ReadStream.open (fs.js:1951:6)
at new ReadStream (fs.js:1938:10)
at Object.fs.createReadStream (fs.js:1885:10)
at Server.<anonymous> (/Users/xpg/project/test/index.js:18:6)
at emitTwo (events.js:106:13)
at Server.emit (events.js:191:7)
at HTTPParser.parserOnIncoming [as onIncoming] (_http_server.js:546:12)
at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)

I just want to know that if I want to pass a Buffer as the path argument, what is the options I should provide, passing 'binary' doesn't work.

I try it with both Node 6.11.0 and Node 8.4.0

See https://stackoverflow.com/questions/13230487/converting-a-buffer-into-a-readablestream-in-nodejs — ivo, Aug 26 '17 at 02:03
A file path has to end up being a string that the OS will accept and can't be random binary so it's unclear what you're trying to do by passing a buffer anyway. If you pass a buffer, it will presumably call `.toString()` on it. — jfriend00, Aug 26 '17 at 03:10
@ivo -That post has nothing to do with the first argument to `fs.createReadStream()`. — jfriend00, Aug 26 '17 at 03:11
If a `Buffer` is passed as argument to `fs.createReadStream()`, it should indicate the file path, not file content. — shaochuancs, Aug 26 '17 at 03:27
@jfriend00 Due to the fact that he's not passing a path and has a buffer of binary data, I believe that he actually wants to create a readable stream from the buffer contents. I see you suggested that in your accepted answer. — ivo, Aug 26 '17 at 12:32

jfriend00 · Accepted Answer · 2020-09-11T21:27:14.133

The first argument to fs.createReadStream() must be the file path. You can apparently pass the path in a Buffer object, but it still must be an acceptable OS path when the Buffer is converted to a string.

It appears you are trying to pass the file content to fs.createReadStream(). That is not how that API works. If you look into the code for fs.createReadStream() it is completely clear in the code that it is going to call fs.open() and pass the first argument from fs.createReadStream() as the file path for fs.open().

If what you're trying to do is to create a readable stream from a buffer (no file involved), then you need to do that a different way. You can see how to do that here in this answer: Converting a Buffer into a ReadableStream in Node.js.

Conceptually, you just create a readable stream object, push the data you already have into it from your Buffer, push a null to signify the end of the stream and create a noop _read() method and you're done. You can then use that readable stream with any other code that expects to read it.

score 17 · Answer 2 · edited Jun 26 '22 at 17:19

Creating a Readable Stream From Buffer.

You can easily create a Readable Stream from a Buffer, however using fs.createReadStream() does indeed require first writing it to a file path.

Using stream.Duplex()

No need to write a local file in order to get a readable stream
Saves I/O and speeds things up.
Works everywhere fs.createReadStream() is desired.
Great for writing to cloud services where local storage my not be feasible.

Example:

const {Duplex} = require('stream'); // Native Node Module 

function bufferToStream(myBuffer) {
    let tmp = new Duplex();
    tmp.push(myBuffer);
    tmp.push(null);
    return tmp;
}

const myReadableStream = bufferToStream(your_buffer);

// use myReadableStream anywhere you would use a stream 
// created using fs.createReadStream('some_path.ext');
// For really large streams, you may want to pipe the buffer into the Duplex.

That's an interesting approach, may be it could be helpful for this issue https://stackoverflow.com/questions/70039150/nodejs-extract-tar-gizp-archive-and-avoid-err-out-of-range?noredirect=1#comment123811531_70039150 — loretoparisi, Nov 19 '21 at 22:03

score 4 · Answer 3 · answered Aug 26 '17 at 03:45

@jfriend00 has already provided a very clear explanation on this issue. If a Buffer object is passed as argument to fs.createReadStream(), it should indicate the file path, not file content. As @Littlee asked in comment, here is an example code:

var express = require('express');
var router = express.Router();
var fs = require('fs')

router.get('/test', function(req, res) {
  var buf = Buffer.from('./test.html');
  fs.createReadStream(buf).pipe(res);
});

Please note the Buffer buf indicates a file path ("./test.html"), not the file test.html's content.

ElcapiJoel · Answer 4 · 2023-08-08T02:46:03.507

If you are using NestJs as Backend, this can be a solution, Good Luck

import { StreamableFile } from '@nestjs/common';

# File is type  (file: Express.Multer.File)

const {originalname, buffer} = file;
const streamableObject =  new StreamableFile(buffer),
const readStream  = streamableObject.stream

Documentation about Streamable Class, in my particular case I used this code to upload files to Google Drive Api, given that Api Google requires a createReadStream() https://docs.nestjs.com/techniques/streaming-files

How to pass a Buffer as argument of fs.createReadStream

4 Answers4

Linked