Circle Piping to and from 2 Python Subprocesses

Question

I needed help regarding the subprocess module. This question might sound repeated, and I have seen a number of articles related to it in a number of ways. But even so I am unable to solve my problem. It goes as follows:
I have a C program 2.c it's contents are as follows:

#include<stdio.h>
int main()
{
int a;
scanf("%d",&a);
while(1)
  {
   if(a==0)               //Specific case for the first input
     {
      printf("%d\n",(a+1));
      break;
     }
   scanf("%d",&a);
   printf("%d\n",a);
  }
return 0;
}

I need to write a python script which first compiles the code using subprocess.call() and then opens two process using Popen to execute the respective C-program. Now the output of the first process must be the input of the second and vice versa. So essentially, if my initial input was 0, then the first process outputs 2, which is taken by second process. It in turn outputs 3 and so on infinitely.

The below script is what I had in mind, but it is flawed. If someone can help me I would very much appreciate it.

from subprocess import *
call(["gcc","2.c"])
a = Popen(["./a.out"],stdin=PIPE,stdout=PIPE) #Initiating Process
a.stdin.write('0')
temp = a.communicate()[0]
print temp
b = Popen(["./a.out"],stdin=PIPE,stdout=PIPE) #The 2 processes in question
c = Popen(["./a.out"],stdin=PIPE,stdout=PIPE)
while True:
    b.stdin.write(str(temp))
    temp = b.communicate()[0]
    print temp
    c.stdin.write(str(temp))
    temp = c.communicate()[0]
    print temp
a.wait()
b.wait()
c.wait()

Answer 1

If you want the output of the first command a to go as the input of the second command b and in turn b's output is a's input—in a circle like a snake eating its tail— then you can't use .communicate() in a loop: .communicate() doesn't return until the process is dead and all the output is consumed.

One solution is to use a named pipe (if open() doesn't block in this case on your system):

#!/usr/bin/env python3
import os
from subprocess import Popen, PIPE

path = 'fifo'
os.mkfifo(path) # create named pipe
try:
    with open(path, 'r+b', 0) as pipe, \
         Popen(['./a.out'], stdin=PIPE, stdout=pipe) as b, \
         Popen(['./a.out'], stdout=b.stdin, stdin=pipe) as a:
        pipe.write(b'10\n') # kick-start it
finally:
    os.remove(path) # clean up

It emulates a < fifo | b > fifo shell command from @alexander barakin answer.

---

Here's more complex solution that funnels the data via the python parent process:

#!/usr/bin/env python3
import shutil
from subprocess import Popen, PIPE

with Popen(['./a.out'], stdin=PIPE, stdout=PIPE, bufsize=0) as b, \
     Popen(['./a.out'], stdout=b.stdin, stdin=PIPE, bufsize=0) as a:
    a.stdin.write(b'10\n') # kick-start it
    shutil.copyfileobj(b.stdout, a.stdin) # copy b's stdout to a' stdin

This code connects a's output to b's input using redirection via OS pipe (as a | b shell command does).

To complete the circle, b's output is copied to a's input in the parent Python code using shutil.copyfileobj().

This code may have buffering issues: there are multiple buffers in between the processes: C stdio buffers, buffers in Python file objects wrapping the pipes (controlled by bufsize).

bufsize=0 turns off the buffering on the Python side and the data is copied as soon as it is available. Beware, bufsize=0 may lead to partial writes—you might need to inline copyfileobj() and call write() again until all read data is written.

Call setvbuf(stdout, (char *) NULL, _IOLBF, 0), to make the stdout line-buffered inside your C program:

#include <stdio.h>

int main(void)
{
  int a;
  setvbuf(stdout, (char *) NULL, _IOLBF, 0); /* make line buffered stdout */
  do {
    scanf("%d",&a);
    printf("%d\n",a-1);
    fprintf(stderr, "%d\n",a); /* for debugging */
  } while(a > 0);
  return 0;
}

Output

10
9
8
7
6
5
4
3
2
1
0
-1

The output is the same.

Due to the way the C child program is written and executed, you might also need to catch and ignore BrokenPipeError exception at the end on a.stdin.write() and/or a.stdin.close() (a process may be already dead while there is uncopied data from b).

Answer 2

Problem is here

while True:
    b.stdin.write(str(temp))
    temp = b.communicate()[0]
    print temp
    c.stdin.write(str(temp))
    temp = c.communicate()[0]
    print temp

Once communicate has returned, it does noting more. You have to run the process again. Plus you don't need 2 processes open at the same time.

Plus the init phase is not different from the running phase, except that you provide the input data.

what you could do to simplify and make it work:

from subprocess import *
call(["gcc","2.c"])
temp = str(0)

while True:
    b = Popen(["./a.out"],stdin=PIPE,stdout=PIPE) #The 2 processes in question
    b.stdin.write(temp)
    temp = b.communicate()[0]
    print temp
    b.wait()

Else, to see 2 processes running in parallel, proving that you can do that, just fix your loop as follows (by moving the Popen calls in the loop):

while True:

    b = Popen(["./a.out"],stdin=PIPE,stdout=PIPE) #The 2 processes in question
    c = Popen(["./a.out"],stdin=PIPE,stdout=PIPE)

    b.stdin.write(str(temp))
    temp = b.communicate()[0]
    print temp
    c.stdin.write(str(temp))
    temp = c.communicate()[0]
    print temp

better yet. b output feeds c input:

while True:

    b = Popen(["./a.out"],stdin=PIPE,stdout=PIPE) #The 2 processes in question
    c = Popen(["./a.out"],stdin=b.stdout,stdout=PIPE)

    b.stdin.write(str(temp))
    temp = c.communicate()[0]
    print temp