Handling chunked responses from process.stdout 'data' event

Question

I have some code which I can't seem to fix. It looks as follows:

var childProcess = require('child_process');
var spawn = childProcess.spawn;
child = spawn('./simulator',[]);

child.stdout.on('data',
    function(data){
        console.log(data);
    }
);

This is all at the backend of my web application which is running a specific type of simulation. The simulator executable is a c program which runs a loop waiting to be passed data (via its standard input) When the inputs come in for the simulation (ie from the client), I parse the input, and then write data to the child process stdin as follows:

child.stdin.write(INPUTS);

Now the data coming back is 40,000 bytes give or take. But the data seems to be getting broken into chunks of 8192 bytes. I've tried fixing the standard output buffer of the c program but it doesnt fix it. I'm wondering if there is a limit to the size of the 'data' event that is imposed by node.js? I need it to come back as one chunk.

Welcome to async. Related: http://stackoverflow.com/q/5083914/112196 — Pero P., Aug 14 '13 at 23:11
Instead of the 'flowing' data, you could try the [read()](http://nodejs.org/api/stream.html#stream_readable_read_size) function. @ppejovic has a point though.. Why would you need all data at once? — Wrikken, Aug 14 '13 at 23:14

score 6 · Accepted Answer · answered Aug 15 '13 at 03:14

6

The buffer chunk sizes are applied in node. Nothing you do outside of node will solve the problem. There is no way to get what you want from node without a little extra work in your messaging protocol. Any message larger than the chunk size will be chunked. There are two ways you can handle this issue.

If you know the total output size before you start to stream out of C, prepend the message length to the data so the node process knows how many chunks to pull before terminating the entire message.
Determine a special character you can append to the message you are sending from the C program. When node sees that character, you end the input from that message.

answered Aug 15 '13 at 03:14

Anthony Hildoer

545
3
6

Thanks Anthony, good to know. As it happens I'm just doing an ugly hack - print to standard error at the end. Seems to be working fine although it does leave me open to something beyond my control causing the print to standard error getting back to the node process before the last print to standard output. Anyway, good to know, thanks. – user1276273 Aug 15 '13 at 06:13
@user1276273 the reason standard error is coming back first is most likely because STDERR is usually blocking IO (the IO completes before anything else happens) whereas STDOUT can be pre-empted. so, if you fire off STDOUT and then immediately fire off STDERR, chances are the IO for STDOUT is still going, and STDERR will get priority and finish, then STDOUT will finish. You will either have to flush STDOUT manually, then fire off the STDERR message, or you will have to put data and control signals all in STDOUT. – Anthony Hildoer Aug 15 '13 at 16:31
Wow ok that is very helpful. It did happen from time to time and I couldn't figure out why. This information is much appreciated. – user1276273 Aug 15 '13 at 20:04

Pero P. · Answer 2 · 2013-08-14T23:56:08.600

5

If you are dealing with IO in a web application you really want to stick with the async methods. You need something like the following (untested). There is a good sample of how to consume the Stream API in the docs

var data = '';
child.stdout.on('data',
    function(chunk){
        data += chunk;
    }
);    

child.stdout.on('end',
    function(){
        // do something with var data
    }
);

edited Aug 14 '13 at 23:56

answered Aug 14 '13 at 23:19

Pero P.

25,813
9
61
85

unfortunately i can't have the c program starting over and over again. It needs to be constantly running, accepting data from it's stdin and printing it to stdout. There is a solution where I print a specific word to cause an "end" of sorts, I just thought there might be an easy fix. – user1276273 Aug 15 '13 at 01:27

score 2 · Answer 3 · answered Feb 11 '16 at 17:44

I ran into the same problem. I tried many different things and was starting to get annoyed. I tried prepending and appending with special characters. Maybe I was stupid but I just couldn't get it right.

I ran into a module called linerstream which basically parses every chunk until it sees an EOF. You can use it like this:

process.stdout.pipe(new Linerstream()).on('data', (data) => {
    // data here is complete and not chunked 
});

The important part is that you do have to write data to stdout with a line that ends with EOF. Otherwise it doesn't know it is the end.

I can say this worked me. Hopefully it helps other people.

score 1 · Answer 4 · answered Aug 15 '13 at 00:54

ppejovic's solution works, but I prefer concat-stream.

var concat = require('concat-stream');
child.stdout.pipe(concat(function(data) {
    // all your data ready to be used.
});

There are a number of good stream helpers worth looking into based on your problem area. Take a look at substack's stream-handbook.

Handling chunked responses from process.stdout 'data' event

4 Answers4

Linked