2

I have a node.js script which starts a python subprocess and reads its stdout. This works as long as the python process does not try to read from stdin. Then the parent process does not get anything from the child.

I have the node.js script and two python test cases here: (both examples work if you comment the lines that try to read from stdin)

First child:

import sys

print('before')

for line in sys.stdin:
    print(line)

print('after')

Second child:

import sys 

print('before')

while True:
    line = sys.stdin.readline()

    if line != '':
        print(line)
    else:
        break

print('after')

Parent:

const spawn = require('child_process').spawn;

let client = spawn('python', ['test1.py'], {cwd: '/tmp'});

client.stdout.on('data', (data) => {
  console.log(data.toString());
});

client.stderr.on('data', (data) => {
  console.log(data.toString());
});

client.on('close', () => {
  console.log('close');
});

client.on('exit', () => {
  console.log('exit');
});


client.on('disconnect', () => {
  console.log('disconnect');
})
Martin
  • 665
  • 1
  • 7
  • 24
  • I don't know node.js but from the python perspective a line is written but since its a pipe, not a tty, it is buffered waiting for more data. You can do `print('before', flush=True)` to send immediately. Then it waits for data and ... well... you need to send data. – tdelaney Nov 24 '16 at 17:12
  • The `flush=True` trick did solve the problem. If you post this as an answer I will accept it :) – Martin Nov 24 '16 at 17:52
  • It might be better to fix this on the `node.js` side with something like `const spawn = require('pty.js').spawn;` This question discusses splitting stdout/err streams http://stackoverflow.com/questions/15339379/node-js-spawning-a-child-process-interactively-with-separate-stdout-and-stderr-s. I'm happy to supply an answer... but I'm not sure its the best answer. – tdelaney Nov 24 '16 at 17:57
  • ...mistyped. that was 'pty.js' – tdelaney Nov 24 '16 at 17:58

1 Answers1

5

A process stdout can be unbuffered, line buffered or block buffered depending on how the process was started. In particular, programs started from the console are line buffered and programs whose stdout is redirected (to a pipe or file) are block buffered. This is done to increase overall program efficiently. People want to see things right away so terminals are line buffered but other programs and files can wait and get things in bigger blocks, so they are block buffered.

You can fix the problem on the python side by forcing the data to be fluxhed on each write. You can do this with the print statement or with the sys.stdout object itself

print('line 1', flush=True)
print('line 2')
print('line 3')
sys.stdout.flush()

You can also fix it on the node.js side by emulating a terminal, basically tricking the program into thinking it is displaying to a user.

const spawn = require('pty.js').spawn;

This is more generic - you don't need the child's cooperation to make it work. But it can get complicated. Some child processes get information about the attached tty to do more complicated things like create menus or color output. But its frequently a great choice.

tdelaney
  • 73,364
  • 6
  • 83
  • 116
  • Thanks for your answer. The pty solution works but I would like to take another look at the first idea. When I flush stdout I can receive the output from python but I have another problem: when I send text to the child process using `client.stdin.write('test')` this does not show up in the python program. Do I have to somehow flush `stdin` as well? – Martin Nov 27 '16 at 16:45
  • I think so. I don't really know node.js but if spawn sets up a pipe to the child stdin, its block buffered and has the same issues. Since you are reading lines, you also need to include a newline. – tdelaney Nov 27 '16 at 16:50