1

I understand you might think this is a duplicated question, but so far I did not find solutions to my own problem.

Currently, I have a C program continuously producing streams of data. I hope there is a way for my python program can read those data so I don't have to finish everything in C.

The C program mainly feels like this:

int i = 0;
while(1){
    printf("%d %d\n", i, i+1);
    i++;
}

I read and tried the subprocess in Python, seems they all need to wait for command to complete.(for example this: Constantly print Subprocess output while process is running) I am hoping some buffer mechanism so I can record this bit stream and process it line by line. Thanks.

Several assumptions:

1) processed line can be discarded

2) buffer/queue size can be no problem since I can control the rate from source to match the processing speed of python program.

A little more background: The C program basicly drives a camera feed (and that's why it is written in C), does some OpenCV stuff and outputs an object position (x, y) int value. The python program needs the position to do some further processing.

Thanks.

Community
  • 1
  • 1
Kamarkaka
  • 87
  • 2
  • 8
  • have you tried [this answer](http://stackoverflow.com/a/4417735/4279) from [the question you've linked](http://stackoverflow.com/questions/4417546/constantly-print-subprocess-output-while-process-is-running)? It is a better analog of [the answer that you've accepted](http://stackoverflow.com/a/20393687/4279). See also [Python: read streaming input from subprocess.communicate()](http://stackoverflow.com/q/2715847/4279) – jfs Dec 06 '13 at 14:48
  • note: if you can modify the C program; you could add `setvbuf(stdout, (char *) NULL, _IOLBF, 0);` (make the program line-buffered, regardless whether it prints to a terminal or another program (pipe)) to get each line immediately as soon as it is printed in C. – jfs Dec 06 '13 at 14:58

1 Answers1

1

Here's a way to process line by line the output of a c (or shell) program with the subprocess module:

from subprocess import Popen, PIPE
# Set up the process
p = Popen("yourprogram", stdout=PIPE, close_fds=True)
count = 0
while True:
    count += 1
    print '%s: %s' % (count, p.stdout.readline().strip())

For demonstration purposes, yourprogram can be a simple shell script (with the executable bit set):

#!/bin/sh
while sleep 1s
do
    date
done

Running the python program produces:

$ python test.py
1: Thu Dec  5 23:31:45 PST 2013
2: Thu Dec  5 23:31:46 PST 2013
3: Thu Dec  5 23:31:47 PST 2013
4: Thu Dec  5 23:31:48 PST 2013
5: Thu Dec  5 23:31:49 PST 2013
6: Thu Dec  5 23:31:50 PST 2013
7: Thu Dec  5 23:31:51 PST 2013

It continues until you terminate it.

John1024
  • 109,961
  • 14
  • 137
  • 171
  • Thanks for your reply but just like others on stackoverflow, your solution still needs "yourCprogram" to complete before reading lines, which is not what I need exactly. – Kamarkaka Dec 05 '13 at 07:48
  • @Kamarkaka That is not correct. In this solution, "yourCprogram" does run in parallel with the python program. The two run independently but are connected by a pipe. – John1024 Dec 05 '13 at 08:13
  • @Kamarkaka The other code that you have looked at probably had wait statements (e.g. `p.wait()`) which forced the python code to wait until "yourCprogram" ended. I didn't include one above which is why the two programs can continue to run in parallel. – John1024 Dec 05 '13 at 18:19
  • I tried your code but it really waits until the C program to finish. If I set the C program to run infinitely, nothing will return forever...I know it's weird, any possible reason you can think of? – Kamarkaka Dec 06 '13 at 07:19
  • @Kamarkaka, sorry about that! In my haste, I assumed that readlines behaved like a generator. It didn't. I have fixed the code now. – John1024 Dec 06 '13 at 07:38
  • I really appreciate your help. I think I am getting closer! Now when I try to use "sh test.sh" in Popen, it seems error... – Kamarkaka Dec 06 '13 at 08:07
  • btw I am using Popen(["test.sh"], ...), error was "no such file or directory"; when I use Popen(['/home/pi/project/test.sh'], ...), error was "access denied" – Kamarkaka Dec 06 '13 at 08:15
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/42633/discussion-between-john1024-and-kamarkaka) – John1024 Dec 06 '13 at 08:47
  • why would you use an infinite loop? Here's [an example that stops reading on EOF](http://stackoverflow.com/a/17698359/4279). – jfs Dec 06 '13 at 14:55