263

I am using a library, ya-csv, that expects either a file or a stream as input, but I have a string.

How do I convert that string into a stream in Node?

Morten Siebuhr
  • 6,068
  • 4
  • 31
  • 43
pathikrit
  • 32,469
  • 37
  • 142
  • 221

12 Answers12

215

As @substack corrected me in #node, the new streams API in Node v10 makes this easier:

const Readable = require('stream').Readable;
const s = new Readable();
s._read = () => {}; // redundant? see update below
s.push('your text here');
s.push(null);

… after which you can freely pipe it or otherwise pass it to your intended consumer.

It's not as clean as the resumer one-liner, but it does avoid the extra dependency.

(Update: in v0.10.26 through v9.2.1 so far, a call to push directly from the REPL prompt will crash with a not implemented exception if you didn't set _read. It won't crash inside a function or a script. If inconsistency makes you nervous, include the noop.)

Community
  • 1
  • 1
Garth Kidd
  • 7,264
  • 5
  • 35
  • 36
  • 6
    From the [docs (link)](http://nodejs.org/api/stream.html#stream_readable_read_size_1): "All Readable stream implementations must provide a `_read` method to fetch data from the underlying resource." – Felix Rabe Jun 11 '14 at 21:38
  • 2
    @eye_mew you need to require('stream') first – Jim Jones Dec 05 '15 at 20:15
  • @GarthKidd I am trying to use this example with a readline interface. [example](https://nodejs.org/api/readline.html#readline_rl_close#apicontent) In my case I need to use the 'line' event and 'close' event. The 'line' event is triggering, but I can't seem to get the 'close' event to fire. Any ideas? – sdc Feb 21 '16 at 10:01
  • If anyone comes across the same issue that I had, I found a solution. From inside the 'line' event I had to manually call `rl.close()` when I came across a given criteria `rl.on('line', (line) => { if(line.indexOf('\n\n') > -1) { rl.close(); } });` – sdc Feb 21 '16 at 10:30
  • 11
    Why do you push `null` into the stream's buffer? – dopatraman Feb 23 '16 at 19:22
  • Also, how would I stream an `Array`, given that a string is just an Array of characters? – dopatraman Feb 23 '16 at 20:45
  • 7
    @dopatraman `null` tells the stream that it has finished reading all the data and to close the stream – chrishiestand Jul 30 '16 at 00:14
  • 3
    Looks like you shouldn’t do it this way. Quoting [the docs](https://nodejs.org/download/nightly/v10.0.0-nightly20180405b29c36b807/docs/api/stream.html#stream_readable_push_chunk_encoding): “The `readable.push()` method is intended be called only by Readable Implementers, and only from within the `readable._read()` method.” – Axel Rauschmayer Apr 06 '18 at 11:16
  • 2
    According to the nodejs docs the example is violating proper usage. i.e. those s.push() calls should be inside of the _read implementation. See the second last comment in the docs https://nodejs.org/api/stream.html#stream_readable_push_chunk_encoding – Trenton D. Adams Nov 20 '18 at 04:18
  • 1
    @TrentonD.Adams it appears that that comment is from [the v10.x docs](https://nodejs.org/docs/latest-v10.x/api/stream.html#stream_readable_push_chunk_encoding). [The v14.x docs](https://nodejs.org/docs/latest-v14.x/api/stream.html#stream_readable_push_chunk_encoding) simply say "The readable.push() method is used to push the content into the internal buffer" which implies to me that it's a perfectly fine method to use. – derpedy-doo Apr 02 '21 at 14:32
  • eslint is complaining about the `Unexpected empty arrow function`. I replaced this with `image._read = () => undefined;` to satisfy it. – Zikoat Mar 01 '22 at 10:37
162

Do not use Jo Liss's resumer answer. It will work in most cases, but in my case it lost me a good 4 or 5 hours bug finding. There is no need for third party modules to do this.

NEW ANSWER:

var Readable = require('stream').Readable

var s = new Readable()
s.push('beep')    // the string you want
s.push(null)      // indicates end-of-file basically - the end of the stream

This should be a fully compliant Readable stream. See here for more info on how to use streams properly.

OLD ANSWER: Just use the native PassThrough stream:

var stream = require("stream")
var a = new stream.PassThrough()
a.write("your string")
a.end()

a.pipe(process.stdout) // piping will work as normal
/*stream.on('data', function(x) {
   // using the 'data' event works too
   console.log('data '+x)
})*/
/*setTimeout(function() {
   // you can even pipe after the scheduler has had time to do other things
   a.pipe(process.stdout) 
},100)*/

a.on('end', function() {
    console.log('ended') // the end event will be called properly
})

Note that the 'close' event is not emitted (which is not required by the stream interfaces).

sziraqui
  • 5,763
  • 3
  • 28
  • 37
B T
  • 57,525
  • 34
  • 189
  • 207
141

From node 10.17, stream.Readable have a from method to easily create streams from any iterable (which includes array literals):

const { Readable } = require("stream")

const readable = Readable.from(["input string"])

readable.on("data", (chunk) => {
  console.log(chunk) // will be called once with `"input string"`
})

Note that at least between 10.17 and 12.3, a string is itself a iterable, so Readable.from("input string") will work, but emit one event per character. Readable.from(["input string"]) will emit one event per item in the array (in this case, one item).

Also note that in later nodes (probably 12.3, since the documentation says the function was changed then), it is no longer necessary to wrap the string in an array.

https://nodejs.org/api/stream.html#stream_stream_readable_from_iterable_options

John
  • 29,546
  • 11
  • 78
  • 79
Fizker
  • 2,434
  • 1
  • 18
  • 12
  • 2
    According to [stream.Readable.from](https://nodejs.org/api/stream.html#stream_stream_readable_from_iterable_options), *Calling Readable.from(string) or Readable.from(buffer) will not have the strings or buffers be iterated to match the other streams semantics for performance reasons.* – abbr Mar 01 '20 at 00:04
  • 1
    My bad. The function was added in 10.7, and behaved the way I originally described. Sometime since, strings no longer need to be wrapped in arrays (since 12.3, it no longer iterates each character individually). – Fizker May 17 '20 at 21:32
35

Just create a new instance of the stream module and customize it according to your needs:

var Stream = require('stream');
var stream = new Stream();

stream.pipe = function(dest) {
  dest.write('your string');
  return dest;
};

stream.pipe(process.stdout); // in this case the terminal, change to ya-csv

or

var Stream = require('stream');
var stream = new Stream();

stream.on('data', function(data) {
  process.stdout.write(data); // change process.stdout to ya-csv
});

stream.emit('data', 'this is my string');
SUDO Los Angeles
  • 1,555
  • 12
  • 10
zemirco
  • 16,171
  • 8
  • 62
  • 96
  • 13
    This code breaks stream conventions. `pipe()` is supposed to return the destination stream, at very least. – greim Jun 30 '14 at 16:02
  • 2
    The end event isn't called if you use this code. This is not a good way to create a stream that can be used generally. – B T Sep 03 '14 at 17:28
12

Edit: Garth's answer is probably better.

My old answer text is preserved below.


To convert a string to a stream, you can use a paused through stream:

through().pause().queue('your string').end()

Example:

var through = require('through')

// Create a paused stream and buffer some data into it:
var stream = through().pause().queue('your string').end()

// Pass stream around:
callback(null, stream)

// Now that a consumer has attached, remember to resume the stream:
stream.resume()
Community
  • 1
  • 1
Jo Liss
  • 30,333
  • 19
  • 121
  • 170
  • I couldn't get zeMirco's solution to work for my use case, but `resumer` worked quite well. Thanks! – mpen Sep 09 '13 at 16:13
  • The @substack resumer suggestion worked very well for me. Thanks! – Garth Kidd Feb 28 '14 at 03:28
  • 2
    Resumer is great, but the "auto-resumes the stream on nextTick" can yield surprises if you expect you can pass the stream to unknown consumers! I had some code that piped a content stream to a file if a db save of metadata succeeded. That was a lurking bug, it happened to succeed when the db write returned success immediately! I later refactored things to be inside an async block, and boom, the stream was never readable. Lesson: if you don't know who's going to consume your stream, stick to the through().pause().queue('string').end() technique. – Jolly Roger Apr 10 '14 at 16:56
  • 2
    I spent about 5 hours debugging my code because I used the resumer part of this answer. It'd be great if you could like.. remove it – B T Jan 22 '15 at 01:53
11

There's a module for that: https://www.npmjs.com/package/string-to-stream

var str = require('string-to-stream')
str('hi there').pipe(process.stdout) // => 'hi there' 
UpTheCreek
  • 31,444
  • 34
  • 152
  • 221
Lori
  • 1,392
  • 1
  • 21
  • 29
  • 1
    Is this a pun on "there's an app for that"? ;) – masterxilo Mar 08 '18 at 09:16
  • 1
    The link in the comment is the useful one: https://www.npmjs.com/package/string-to-stream – Dem Pilafian Jun 13 '18 at 07:00
  • FYI I tried to use this library for writing JSON to google drive but it wouldn't work for me. Wrote an article about this here: https://medium.com/@dupski/nodejs-creating-a-readable-stream-from-a-string-e0568597387f . Also added as an answer below – Russell Briggs Jul 14 '19 at 06:14
9

Another solution is passing the read function to the constructor of Readable (cf doc stream readeable options)

var s = new Readable({read(size) {
    this.push("your string here")
    this.push(null)
  }});

you can after use s.pipe for exemple

Philippe T.
  • 1,182
  • 7
  • 11
6

in coffee-script:

class StringStream extends Readable
  constructor: (@str) ->
    super()

  _read: (size) ->
    @push @str
    @push null

use it:

new StringStream('text here').pipe(stream1).pipe(stream2)
xinthink
  • 1,460
  • 1
  • 18
  • 22
5

I got tired of having to re-learn this every six months, so I just published an npm module to abstract away the implementation details:

https://www.npmjs.com/package/streamify-string

This is the core of the module:

const Readable = require('stream').Readable;
const util     = require('util');

function Streamify(str, options) {

  if (! (this instanceof Streamify)) {
    return new Streamify(str, options);
  }

  Readable.call(this, options);
  this.str = str;
}

util.inherits(Streamify, Readable);

Streamify.prototype._read = function (size) {

  var chunk = this.str.slice(0, size);

  if (chunk) {
    this.str = this.str.slice(size);
    this.push(chunk);
  }

  else {
    this.push(null);
  }

};

module.exports = Streamify;

str is the string that must be passed to the constructor upon invokation, and will be outputted by the stream as data. options are the typical options that may be passed to a stream, per the documentation.

According to Travis CI, it should be compatible with most versions of node.

Chris Allen Lane
  • 6,334
  • 5
  • 25
  • 31
5

Heres a tidy solution in TypeScript:

import { Readable } from 'stream'

class ReadableString extends Readable {
    private sent = false

    constructor(
        private str: string
    ) {
        super();
    }

    _read() {
        if (!this.sent) {
            this.push(Buffer.from(this.str));
            this.sent = true
        }
        else {
            this.push(null)
        }
    }
}

const stringStream = new ReadableString('string to be streamed...')
Russell Briggs
  • 993
  • 8
  • 8
2

In a NodeJS, you can create a readable stream in a few ways:

SOLUTION 1

You can do it with fs module. The function fs.createReadStream() allows you to open up a readable stream and all you have to do is pass the path of the file to start streaming in.

const fs = require('fs');

const readable_stream = fs.createReadStream('file_path');

SOLUTION 2

If you don't want to create file, you can create an in-memory stream and do something with it (for example, upload it somewhere). ​You can do this with stream module. You can import Readable from stream module and you can create a readable stream. When creating an object, you can also implement read() method which is used to read the data out of the internal buffer. If no data available to be read, null is returned. The optional size argument specifies a specific number of bytes to read. If the size argument is not specified, all of the data contained in the internal buffer will be returned.

const Readable = require('stream').Readable;

const readable_stream = new Readable({
  ​read(size) {
   ​// ...
​  }
});

SOLUTION 3

When you are fetching something over the network, that can be fetched like stream (for example you are fetching a PDF document from some API).

const axios = require('axios');

const readable_stream = await axios({
  method: 'get',
  url: "pdf_resource_url",
  responseType: 'stream'
}).data;

SOLUTION 4

Third party packages can support creating of streams as a feature. That is a way with aws-sdk package that is usually used for uploading files to S3.

const file = await s3.getObject(params).createReadStream();
NeNaD
  • 18,172
  • 8
  • 47
  • 89
  • 4
    These solutions explain various ways to create a stream, but none of them the question, which is asking how to convert a *string* into a stream. – cjol Jan 10 '22 at 14:54
  • Maybe, but it still helped me to fix my own (similar) problem. – raarts Jan 23 '22 at 10:16
0

JavaScript is duck-typed, so if you just copy a readable stream's API, it'll work just fine. In fact, you can probably not implement most of those methods or just leave them as stubs; all you'll need to implement is what the library uses. You can use Node's pre-built EventEmitter class to deal with events, too, so you don't have to implement addListener and such yourself.

Here's how you might implement it in CoffeeScript:

class StringStream extends require('events').EventEmitter
  constructor: (@string) -> super()

  readable: true
  writable: false

  setEncoding: -> throw 'not implemented'
  pause: ->    # nothing to do
  resume: ->   # nothing to do
  destroy: ->  # nothing to do
  pipe: -> throw 'not implemented'

  send: ->
    @emit 'data', @string
    @emit 'end'

Then you could use it like so:

stream = new StringStream someString
doSomethingWith stream
stream.send()
icktoofay
  • 126,289
  • 21
  • 250
  • 231
  • I get this: `TypeError: string is not a function at String.CALL_NON_FUNCTION (native)` when I use it like `new StringStream(str).send()` – pathikrit Oct 06 '12 at 02:24
  • Just because JavaScript uses duck typing doesn't mean you should reinvent the wheel. Node already provides an implementation for streams. Just create a new instance of `stream.Readable` like @Garth Kidd suggested. – Sukima Oct 02 '14 at 03:15
  • 5
    @Sukima: `stream.Readable` [did not exist](https://web.archive.org/web/20121007204945/http://nodejs.org/api/stream.html) when I wrote this answer. – icktoofay Oct 02 '14 at 04:00