Disordered output while executing multi-processed script using popen

Question

I have been trying to execute a multi-processed python program using popen command in php.

The problem is, output order is not as expected. The PHP code is

if ( ($fp = popen("python multi-processed.py 2>&1", "r")) )
{
    while( !feof($fp) )
    {
        print fread($fp, 1024);
        ob_flush();
        flush();
    }
    fclose($fp);
}
else
    print "can't execute command";

But when I run this command in shell, I am getting output in expected order

python multi-processed.py 2>&1

Then I thought of pipe buffering, and wrote below code in C

FILE *fp = popen("python multi-processed.py -u", "r");
if(fp != NULL)
{
    setvbuf(fp, 0, _IONBF, 0);

    while (!feof(fp))
    {
        fgets(buffer, 1024, fp);
        puts(buffer);
        fflush(fp);
    }
    fclose(fp);
}

Here I am turning off pipe buffering by calling setvbuf(fp, 0, _IONBF, 0);, yet the output order is not as expected.

Edit: Added python file : multi-processed.py

def foo():
    print 'in foo'
    sleep(2)
    print 'in foo after sleep'

def fun():
    print 'in fun'
    sleep(2)
    print 'in fun after sleep'

if __name__ == '__main__':
    p1 = Process(target=foo, args=())
    p2 = Process(target=fun, args=())
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Expected Output:

in foo
in fun
in foo after sleep
in fun after sleep

Actual output:

in foo
in foo after sleep
in fun
in fun after sleep

Can you please help me out in this?

PS: I'm using windows machine, can't run commands like unbuffer

A "multi-processed" program? Or multi-threaded? – deviantfan Jul 03 '14 at 11:22 — deviantfan, Jul 03 '14 at 11:22
multi-processed program @deviantfan – Akshay Jul 03 '14 at 11:23 — Akshay, Jul 03 '14 at 11:23

score 0 · Accepted Answer · edited May 23 '17 at 12:29

Answering my question:

The problem was stdout buffering. It can be corrected by flushing each write to stdout.

From https://stackoverflow.com/a/107717/1519642 , adding below code solved the problem

import sys

class Unbuffered(object):
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
   def __getattr__(self, attr):
       return getattr(self.stream, attr)

sys.stdout = Unbuffered(sys.stdout)

Disordered output while executing multi-processed script using popen

1 Answers1