How to capture non-blocking realtime output from fio using Python

Question

>>>x = subprocess.Popen(["fio", "--filename=/dev/sdd", "--name=job1", "--numjobs=2"], stdout=subprocess.PIPE)

But I need to capture the fio progess in realtime. I'm though of doing a threaded subprocess.stdout.readline, but it doesn't seem to update the performance data:

>>> x = subprocess.Popen(["fio", "--filename=/dev/sdd", "--name=job1", "--numjobs=2"], stdout=subprocess.PIPE)

>>> x.stdout.readline()  
'job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1\n'

>>> x.stdout.readline()  
'...\n'

>>> x.stdout.readline()  
'fio-2.2.11\n'

>>> x.stdout.readline()  
'Starting 2 processes\n'

>>> x.stdout.readline()

it just hangs after this.

Command line execution of fio looks like this:

[root@goblinbank tmp]# fio --filename=/dev/sdd --name=job1 --numjobs=2
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
...
fio-2.2.11
Starting 2 processes
Jobs: 2 (f=2): [R(2)] [0.2% done] [246.3MB/0KB/0KB /s] [63.5K/0/0 iops] [eta 02h:26m:09s]

The text after "Jobs: 2 (f=2): [R(2)]" keeps refreshing with new values.

I ran into this exact same problem, and found a solution, posted answer to a related question: http://stackoverflow.com/a/35052424/432509 Its quite involved to do on both windows and posix systems however. — ideasman42, Jan 30 '16 at 01:20
This may be a duplicate http://stackoverflow.com/questions/375427 — ideasman42, Jan 30 '16 at 01:23
@ideasman, I have seen that question. This is not a duplicate, as the solution described there doesn't work for me. The last line is something i just cannot read into. Doesn't has anything to do with non-blocking at this point I think — Hec, Jan 30 '16 at 01:37
does `fio` print anything on stderr? Have you tried to provide a pseudo-tty e.g., [using `pexpect` module](http://stackoverflow.com/a/25945031/4279)? See also, [Python subprocess readlines() hangs](http://stackoverflow.com/q/12419198/4279) — jfs, Jan 30 '16 at 12:33
@J.F.Sebastian pexpect looks promising, but how do I capture data from the output variable? The readline() on that is blocking too. As long as i can capture each line, parse it for updating some class variable and do that for each line till program runs, I'll be good — Hec, Jan 31 '16 at 04:14
@Hec [read the docs](http://pexpect.readthedocs.org/en/latest/overview.html) and try a few simple examples, to learn how to use `pexpect`. Namely, `.expect()` method and `.before`/`.after` attributes could be useful in your case. Ask; if you a specific problem. — jfs, Jan 31 '16 at 08:31
@J.F.Sebastian Finally didn't need to go that route, turns out there is a flag on fio called --eta=always, and with that set the stdout.readline() doesn't hang anymore! Others reading, don't forget to set universal_newlines=True in Popen. — Hec, Jan 31 '16 at 13:01
@Hec it looks like [`apt-get`s `--quiet` solution I've link above](http://stackoverflow.com/a/25945031/4279). If you can fix it using a command-line option then it is preferable compared to trying to emulate a terminal, to get the necessary output. Don't forget to accept your answer. — jfs, Jan 31 '16 at 14:19
Will do! And thanks for the explanation, and taking the time to help with it. Really appreciate it! — Hec, Jan 31 '16 at 22:51

score 3 · Accepted Answer · edited Oct 30 '16 at 23:29

3

This fixes the issue:

x = subprocess.Popen(["fio", "--filename=/dev/sdd", "--name=job1", "--numjobs=2", "--eta=always"], stdout=subprocess.PIPE, universal_newlines=True)

Notice the --eta=always flag in fio, and the universal_newlines=True argument in Popen.

More info here.

edited Oct 30 '16 at 23:29

Tom Fuller

5,291
7
33
42

answered Jan 31 '16 at 13:04

Hec

824
1
5
24

Álvaro Justen · Answer 2 · 2016-01-30T04:30:53.693

You can then use fcntl to set it as non-blocking IO and it will raise an exception if data is not available to read. Just add:

fcntl.fcntl(process.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

after process creation and verify if there is something to read with a try/except IOError clause, like this:

import fcntl
import os
import shlex
import subprocess
import time

cmd = 'fio --filename=/home/turicas/data/test.tmp --name=job1 --numjobs=2'
process = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE)
fcntl.fcntl(process.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

data = None
while data != '':
    try:
        data = process.stdout.readline()
    except IOError:
        print 'no data. waiting...'
        time.sleep(0.1)  # wait stdout output to be available
    else:
        print 'data read:', data
print 'stdout closed'

doesn't work, can get everything till the reprint (last) line, but reading even a byte into it causes a hang — Hec, Jan 30 '16 at 01:35
No luck still, just get 'no data. waiting...' till execution ends. pexpect as @J.F. Sebastian suggested captures the output, working on filtering and parsing it in a non-blocking way. — Hec, Jan 31 '16 at 04:21

How to capture non-blocking realtime output from fio using Python

2 Answers2