Getting realtime output from ffmpeg to be used in progress bar (PyQt4, stdout)

Question

I've looked at a number of questions but still can't quite figure this out. I'm using PyQt, and am hoping to run ffmpeg -i file.mp4 file.avi and get the output as it streams so I can create a progress bar.

I've looked at these questions: Can ffmpeg show a progress bar? catching stdout in realtime from subprocess

I'm able to see the output of a rsync command, using this code:

import subprocess, time, os, sys

cmd = "rsync -vaz -P source/ dest/"
p, line = True, 'start'


p = subprocess.Popen(cmd,
                     shell=True,
                     bufsize=64,
                     stdin=subprocess.PIPE,
                     stderr=subprocess.PIPE,
                     stdout=subprocess.PIPE)

for line in p.stdout:
    print("OUTPUT>>> " + str(line.rstrip()))
    p.stdout.flush()

But when I change the command to ffmpeg -i file.mp4 file.avi I receive no output. I'm guessing this has something to do with stdout / output buffering, but I'm stuck as to how to read the line that looks like

frame=   51 fps= 27 q=31.0 Lsize=     769kB time=2.04 bitrate=3092.8kbits/s

Which I could use to figure out progress.

Can someone show me an example of how to get this info from ffmpeg into python, with or without the use of PyQt (if possible)

---

EDIT: I ended up going with jlp's solution, my code looked like this:

#!/usr/bin/python
import pexpect

cmd = 'ffmpeg -i file.MTS file.avi'
thread = pexpect.spawn(cmd)
print "started %s" % cmd
cpl = thread.compile_pattern_list([
    pexpect.EOF,
    "frame= *\d+",
    '(.+)'
])
while True:
    i = thread.expect_list(cpl, timeout=None)
    if i == 0: # EOF
        print "the sub process exited"
        break
    elif i == 1:
        frame_number = thread.match.group(0)
        print frame_number
        thread.close
    elif i == 2:
        #unknown_line = thread.match.group(0)
        #print unknown_line
        pass

Which gives this output:

started ffmpeg -i file.MTS file.avi
frame=   13
frame=   31
frame=   48
frame=   64
frame=   80
frame=   97
frame=  115
frame=  133
frame=  152
frame=  170
frame=  188
frame=  205
frame=  220
frame=  226
the sub process exited

Perfect!

Answer 1

In this specific case for capturing ffmpeg's status output (which goes to STDERR), this SO question solved it for me: FFMPEG and Pythons subprocess

The trick is to add universal_newlines=True to the subprocess.Popen() call, because ffmpeg's output is in fact unbuffered but comes with newline-characters.

cmd = "ffmpeg -i in.mp4 -y out.avi"
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,universal_newlines=True)
for line in process.stdout:
    print(line)

Also note that in this code sample the STDERR status output is directly redirected to subprocess.STDOUT

Answer 2

The only way I've found to get dynamic feedback/output from a child process is to use something like pexpect:

#! /usr/bin/python

import pexpect

cmd = "foo.sh"
thread = pexpect.spawn(cmd)
print "started %s" % cmd
cpl = thread.compile_pattern_list([pexpect.EOF,
                                   'waited (\d+)'])
while True:
    i = thread.expect_list(cpl, timeout=None)
    if i == 0: # EOF
        print "the sub process exited"
        break
    elif i == 1:
        waited_time = thread.match.group(1)
        print "the sub process waited %d seconds" % int(waited_time)
thread.close()

the called sub process foo.sh just waits a random amount of time between 10 and 20 seconds, here's the code for it:

#! /bin/sh

n=5
while [ $n -gt 0 ]; do
    ns=`date +%N`
    p=`expr $ns % 10 + 10`
    sleep $p
    echo waited $p
    n=`expr $n - 1`
done

You'll want to use some regular expression that matches the output you're getting from ffmpeg and does some kind of calculation on it to show the progress bar, but this will at least get you the unbuffered output from ffmpeg.

Answer 3

I wrote a dedicated package that gives you a generator function for ffmpeg progress in Python: ffmpeg-progress-yield.

Simply run:

pip3 install ffmpeg-progress-yield

Then, simply do:

from ffmpeg_progress_yield import FfmpegProgress

cmd = [
    "ffmpeg", "-i", "test/test.mp4", "-c:v", "libx264", "-vf", "scale=1920x1080", "-preset", "fast", "-f", "null", "/dev/null",
]

ff = FfmpegProgress(cmd)
for progress in ff.run_command_with_progress():
    print(f"{progress}/100")

Note that this only works for input files where the duration is known in advance.

Answer 4

1. Calling from the shell is generally not required.
2. I know from experince that part of the ffmpeg output comes on stderr, not stdout.

If all you want to do is print the output line, like in your example above, then simply this will do:

import subprocess

cmd = 'ffmpeg -i file.mp4 file.avi'
args = cmd.split()

p = subprocess.Popen(args)

Note that the line of ffmpeg chat is terminated with \r, so it will overwrite in the same line! I think this means you can't iterate over the lines in p.stderr, as you do with your rsync example. To build your own progress bar, then, you may need to handle the reading yourself, this should get you started:

p = subprocess.Popen(args, stderr=subprocess.PIPE)

while True:
  chatter = p.stderr.read(1024)
  print("OUTPUT>>> " + chatter.rstrip())

Answer 5

This answers didn't worked for me :/ Here is the way I did it.

Its from my project KoalaBeatzHunter.

Enjoy!

def convertMp4ToMp3(mp4f, mp3f, odir, kbps, callback=None, efsize=None):
    """
    mp4f:     mp4 file
    mp3f:     mp3 file
    odir:     output directory
    kbps:     quality in kbps, ex: 320000
    callback: callback() to recieve progress
    efsize:   estimated file size, if there is will callback() with %
    Important:
    communicate() blocks until the child process returns, so the rest of the lines 
    in your loop will only get executed after the child process has finished running. 
    Reading from stderr will block too, unless you read character by character like here.
    """
    cmdf = "ffmpeg -i "+ odir+mp4f +" -f mp3 -ab "+ str(kbps) +" -vn "+ odir+mp3f
    lineAfterCarriage = ''

    print deleteFile(odir + mp3f)

    child = subprocess.Popen(cmdf, shell=True, stderr=subprocess.PIPE)

    while True:
        char = child.stderr.read(1)
        if char == '' and child.poll() != None:
            break
        if char != '':
            # simple print to console
#             sys.stdout.write(char)
#             sys.stdout.flush()
            lineAfterCarriage += char
            if char == '\r':
                if callback:
                    size = int(extractFFmpegFileSize(lineAfterCarriage)[0])
                    # kb to bytes
                    size *= 1024
                    if efsize:
                        callback(size, efsize)
                lineAfterCarriage = ''

Next, you need 3 more functions to implement it.

def executeShellCommand(cmd):
    p = Popen(cmd , shell=True, stdout=PIPE, stderr=PIPE)
    out, err = p.communicate()
    return out.rstrip(), err.rstrip(), p.returncode

def getFFmpegFileDurationInSeconds(filename):
    cmd = "ffmpeg -i "+ filename +" 2>&1 | grep 'Duration' | cut -d ' ' -f 4 | sed s/,//"
    time = executeShellCommand(cmd)[0]
    h = int(time[0:2])
    m = int(time[3:5])
    s = int(time[6:8])
    ms = int(time[9:11])
    ts = (h * 60 * 60) + (m * 60) + s + (ms/60)
    return ts

def estimateFFmpegMp4toMp3NewFileSizeInBytes(duration, kbps):
    """
    * Very close but not exact.
    duration: current file duration in seconds
    kbps: quality in kbps, ex: 320000
    Ex:
        estim.:    12,200,000
        real:      12,215,118
    """
    return ((kbps * duration) / 8)

And finally you do:

# get new mp3 estimated size
secs = utls.getFFmpegFileDurationInSeconds(filename)
efsize = utls.estimateFFmpegMp4toMp3NewFileSizeInBytes(secs, 320000)
print efsize

utls.convertMp4ToMp3("AwesomeKoalaBeat.mp4", "AwesomeKoalaBeat.mp3",
                "../../tmp/", 320000, utls.callbackPrint, efsize)

Hope this will help!

Answer 6

You can also do it pretty clearly with PyQt4's QProcess (as asked in the original question) by connecting a slot from the QProcess to a QTextEdit or whatever. I'm still pretty new to python and pyqt but here's how I just managed to do it:

import sys
from PyQt4 import QtCore, QtGui

class ffmpegBatch(QtGui.QWidget):
    def __init__(self):
        super(ffmpegBatch, self).__init__()
        self.initUI()

    def initUI(self):
        layout = QtGui.QVBoxLayout()
        self.edit = QtGui.QTextEdit()
        self.edit.setGeometry(300, 300, 300, 300)
        run = QtGui.QPushButton("Run process")

        layout.addWidget(self.edit)
        layout.addWidget(run)

        self.setLayout(layout)

        run.clicked.connect(self.run)

    def run(self):
        # your commandline whatnot here, I just used this for demonstration
        cmd = "systeminfo"

        proc = QtCore.QProcess(self)
        proc.setProcessChannelMode(proc.MergedChannels)
        proc.start(cmd)
        proc.readyReadStandardOutput.connect(lambda: self.readStdOutput(proc))


    def readStdOutput(self, proc):
        self.edit.append(QtCore.QString(proc.readAllStandardOutput()))

def main():
    app = QtGui.QApplication(sys.argv)
    ex = ffmpegBatch()
    ex.show()
    sys.exit(app.exec_())

if __name__ == '__main__':
    main()

Answer 7

If you have the duration (Which you can also get from the FFMPEG output) you can calculate the progress by reading the elapsed time (time) output when encoding.

A simple example:

  pipe = subprocess.Popen(
        cmd,
        stderr=subprocess.PIPE,
        close_fds=True
  )
  fcntl.fcntl(
        pipe.stderr.fileno(),
        fcntl.F_SETFL,
        fcntl.fcntl(pipe.stderr.fileno(), fcntl.F_GETFL) | os.O_NONBLOCK,
  )
   while True:
            readx = select.select([pipe.stderr.fileno()], [], [])[0]

            if readx: 
                chunk = pipe.stderr.read()

                if not chunk:
                    break

                result = re.search(r'\stime=(?P<time>\S+) ', chunk)
                elapsed_time = float(result.groupdict()['time'])

                # Assuming you have the duration in seconds
                progress = (elapsed_time / duration) * 100

                # Do something with progress here
                callback(progress)

        time.sleep(10)