2

I want to execute an executable of a Fortran code using python, with printing live output. I used subprocess.Popen to see live output as shown here. I do get live output when I execute commands like 'du', however I get output for my Fortran's executable only after finishing its run.

Relevant part of my python script is:

import subprocess as s
a=s.Popen('./fortran_with_fft.exe', shell=True, stdout=s.PIPE)
for line in iter(a.stdout.readline, ''):
    print line

When I run the executable from terminal, it runs without any error and producing the intended output. The same does not happen while running it from python. The Fortran code uses fftw to perform fft calculations using multiple threads. I use 16 threads for this. Relevant part of my Fortran code is: 

nthreads=16
CALL sfftw_init_threads
CALL sfftw_plan_with_nthreads(nthreads)
CALL sfftw_plan_dft_3d(plan,ngrid,ngrid,ngrid,delta,delta,FFTW_BACKWARD,FFTW_estimate)
delta=CMPLX(densitycontr,0.0)
CALL sfftw_execute(plan)

I suspect that this problem of not printing live output is related with the fact that the executable uses multiple threads via fft. Is there any way to get live output for executing such processes which use multiple threads, through python?

Community
  • 1
  • 1
Mihir Kulkarni
  • 21
  • 2
  • how long is your output? It might just get buffered, when operating system detects, that writing is into pipe. – m.wasowski Oct 09 '14 at 12:03
  • Executing the code takes nearly 5 minutes and it keeps printing something from the start while running from the terminal. For python, it prints everything after finishing the run. I can see from 'top' that the process is being executed, even when it is not printing anything. – Mihir Kulkarni Oct 09 '14 at 12:07
  • its output may just land in pipe's buffer. see http://stackoverflow.com/questions/1410849/bypassing-buffering-of-subprocess-output-with-popen-in-c-or-python – m.wasowski Oct 09 '14 at 12:09
  • My output has quite a few number of lines and this doesn't seem to be a buffer issue and I have tried that solution ('-u') already. – Mihir Kulkarni Oct 09 '14 at 12:20
  • is the output separated by newline characters? If not I guess readline would keep reading until it found one. – GP89 Oct 09 '14 at 12:21
  • rather than trying to read it that way you could poll its output using https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate and then print that? – James Kent Oct 09 '14 at 12:24

1 Answers1

1

The following runnable code creates a subprocess and prints the time it takes for each line of output. This proves that we can output subproc lines of data as they arrive.

I don't know why the Fortran program is not working similarly. It shouldn't matter, nor should it matter if a program uses threads or not.

source

#!/usr/bin/env python

'''
pping2.py -- run subprocess, process output as it arrives
'''

# adapted from http://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line

import subprocess, time

def test_ping():
    proc = subprocess.Popen(
        'ping -c5 8.8.8.8', 
        shell=True,
        stdout=subprocess.PIPE,
    )
    start = time.time()
    for line in iter(proc.stdout.readline, ''):
        print '{:.2f} {}'.format(time.time()-start, line.rstrip()) 

if __name__=='__main__':
    test_ping()

output

0.04 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
0.04 64 bytes from 8.8.8.8: icmp_seq=1 ttl=44 time=37.4 ms
1.04 64 bytes from 8.8.8.8: icmp_seq=2 ttl=44 time=36.1 ms
2.04 64 bytes from 8.8.8.8: icmp_seq=3 ttl=44 time=37.2 ms
3.04 64 bytes from 8.8.8.8: icmp_seq=4 ttl=44 time=36.0 ms
4.04 64 bytes from 8.8.8.8: icmp_seq=5 ttl=44 time=36.2 ms
4.04 
4.04 --- 8.8.8.8 ping statistics ---
4.04 5 packets transmitted, 5 received, 0% packet loss, time 4005ms
4.04 rtt min/avg/max/mdev = 36.080/36.629/37.417/0.609 ms
johntellsall
  • 14,394
  • 4
  • 46
  • 40