Questions related to Node.js streams and I/O operations.
Questions marked with this tag are about Node.js streams to handle I/O operations.
Questions tagged [node-streams]
400 questions
155
votes
14 answers
Pipe a stream to s3.upload()
I'm currently making use of a node.js plugin called s3-upload-stream to stream very large files to Amazon S3. It uses the multipart API and for the most part it works very well.
However, this module is showing its age and I've already had to make…
womp
- 115,835
- 26
- 236
- 269
24
votes
4 answers
_read() is not implemented on Readable stream
This question is how to really implement the read method of a readable stream.
I have this implementation of a Readable stream:
import {Readable} from "stream";
this.readableStream = new Readable();
I am getting this error
events.js:136
…
Alexander Mills
- 90,741
- 139
- 482
- 817
23
votes
4 answers
How to pipe multiple readable streams, from multiple api requests, to a single writeable stream?
- Desired Behaviour
- Actual Behaviour
- What I've Tried
- Steps To Reproduce
- Research
Desired Behaviour
Pipe multiple readable streams, received from multiple api requests, to a single writeable stream.
The api responses are from…
user1063287
- 10,265
- 25
- 122
- 218
15
votes
1 answer
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close in Node Pipeline stream
I am using the stream.pipeline functionality from Node to upload some data to S3. The basic idea I'm implementing is pulling files from a request and writing them to S3. I have one pipeline that pulls zip files and writes them to S3 successfully.…
Jason Cromer
- 1,468
- 3
- 11
- 21
15
votes
2 answers
wait for all streams to finish - stream a directory of files
I'm using client.upload in pkgcloud to upload a directory of files. How can I execute a callback after all the streams have finished? Is there a built-in way to register each stream's "finish" event and execute a callback after they have all…
berg
- 614
- 3
- 9
- 23
14
votes
3 answers
Piping requests using gaxios (or axios)
At present I'm performing the trick of piping a request req to a destination url, and piping the response back to res, like so:
const request = require('request');
const url = 'http://some.url.com' + req.originalUrl;
const destination =…
drmrbrewer
- 11,491
- 21
- 85
- 181
14
votes
1 answer
why does attempting to write a large file cause js heap to run out of memory
this code
const file = require("fs").createWriteStream("./test.dat");
for(var i = 0; i < 1e7; i++){
file.write("a");
}
gives this error message after running for about 30 seconds
<--- Last few GCs --->
[47234:0x103001400] 27539 ms:…
schu34
- 965
- 7
- 25
11
votes
1 answer
Typescript error converting a native fetch body webstream to a node stream
package.json
{
"type": "module",
"dependencies": {
"@types/node": "^18.6.5",
"typescript": "^4.7.4"
}
}
tsconfig.json
{
"compilerOptions": {
"target": "ES2021",
"module": "ESNext",
"moduleResolution": "node",
…
Dawei67
- 191
- 1
- 6
11
votes
1 answer
How to create a readstream with a buffer using Node.js
I have this json:
var myJSON = '{"kind": "person", "fullName": "Rivka3"}';
I'm trying to uploed it to bigquery, using createReadStream.
when I save it localy I succeed:
fs.writeFile("/tmp/bq_json_file_new.json", myJSON,…
dina
- 4,039
- 6
- 39
- 67
10
votes
1 answer
How to defer stream read invocation
I'm still trying to grok my way through streams in general. I have been able to stream a large file using multiparty from within form.on('part'). But I need to defer the invocation and resolve the stream before it's read. I have tried PassThrough,…
garajo
- 736
- 4
- 19
9
votes
1 answer
Node.js pass text as stdin of `spawnSync`
I'd think this would be simple, but the following does not work as expected.
I want to pipe data to a process, say (just an arbitrary command for illustration) wc, from Node.
The docs and other SO questions seem to indicate that passing a Stream…
Brian M. Hunt
- 81,008
- 74
- 230
- 343
8
votes
1 answer
Node js Streams: 'close' vs 'finish' event
What is the difference between 'close' and 'finish' events for NodeJS Writable stream?
If we suppose that we have a writable stream that write to disk, are both 'close' and 'finish' events activating after the data is persisted to the disk or not?
Nexx
- 327
- 4
- 4
8
votes
1 answer
How do I close stdin in Node?
I am learning Node.js and thought I had a very simple script but no about of tweaking will ever get my script from hanging forever.
Say I have a dumb server running:
$ nc -l 32001 <
Sukima
- 9,965
- 3
- 46
- 60
8
votes
1 answer
Piping data to writable stream that is not ready to receive data yet
Is there a way to connect a readable stream to a writable stream in Node.js, where the writable is not ready to receive data yet? In other words, I'd like to connect the readable with the writable, but I want to initialize the writable, including…
Alexander Mills
- 90,741
- 139
- 482
- 817
7
votes
1 answer
convert readable stream to save it as a file in local
I am using ssh2-sftp-client to get the file from remote server. I am getting the file in readable stream. I want to convert this readable stream to the desired file (sample.png as a png file, sample.doc file as doc file etc.)
Here is my code-
let…
Komal Bansal
- 789
- 2
- 7
- 20