Problem while reading stream from command line

Question

I want to read the output of the command line while running an object detection algorithm to actively monitor for a certain object. I tried this:

import subprocess 
cmd = './darknet detector test data/obj.data cfg/yolov3_testing.cfg yolov3_training_2.weights data/fire.jpg -thresh 0.02'
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

while proc.poll() is None:
    output = proc.stdout.readline() #get line from output
    out = output.decode('UTF-8')    #convert output from type byte to string
    print(out)

Which kinda works but after reading a hundred or so lines and outputting them, it gets stuck and stops printing lines or executing any code in the loop. Also, most of the lines are read as errors (go to sterr instead of stdout). When terminating darknet, the rest of the lines are immediately printed. This makes me think that while the loop is stuck, the results of the code are actually being buffered, then executed on termination.
Here is the output before terminating darknet (without piping stderr):

And after terminating darknet: Other methods I tried are:

from subprocess import Popen, PIPE
cmd = './darknet detector test data/obj.data cfg/yolov3_testing.cfg yolov3_training_2.weights data/fire.jpg -thresh 0.02'

with Popen(["./darknet", "detector", "test", "data/obj.data", "cfg/yolov3_testing.cfg", "yolov3_training_2.weights", "data/fire.jpg", "-thresh", "0.01"], stdout=PIPE, stderr=PIPE, bufsize=2, universal_newlines=True) as p:
    for line in p.stdout:
        print(line, end='')

and

subprocess.check_call(["./darknet", "detector", "test", "data/obj.data", "cfg/yolov3_testing.cfg", "yolov3_training_2.weights", "data/fire.jpg", "-t

These all have the same problem.

If you aren't interested in error messages try `stderr=subprocess.DEVNULL` instead. — Michael Butscher, Feb 21 '21 at 23:28
I don't think this is a problem with your script. `darknet` just seems to wait until just before it exits to write those last few lines. What happens if you run the same `darknet` command in the terminal, with the terminal itself as standard output? — chepner, Feb 21 '21 at 23:30
As @Programmer said, maybe the output of your python script is being buffered. Or maybe darknet is buffering its output. Check [this question](https://stackoverflow.com/questions/57829831/how-to-disable-stdout-buffer-when-running-shell) to see if you can find something helpful. There are also wrappers or bindings that allow you to call darknet directly from python. — Hernán Alarcón, Feb 21 '21 at 23:31
@HernánAlarcón I have tried what Programmer suggested but it didn't work. Darknet shouldnt be buffering the result chepner also suggested that, but when I run it directly through the terminal it works fine, doesnt get stuck. Ill check out that post but Im having a hard time understanding some of it because I'm completely new to python — Ben van der Colff, Feb 22 '21 at 00:08

Stéphane · Answer 1 · 2021-02-22T21:53:48.117

This isn't the right way to do it. You don't want to repeatedly call the darknet executable for each frame or each image. Instead, you want to use the darknet API to load the neural network into memory once, and then repeatedly invoke detect() using each image or video frame.

However, I understand this doesn't really answer your original question.

If I had to guess: the first flurry of lines you are reading are when the network loads. It takes several seconds for the weights to be loaded and initialized. And then -- especially if you are running on CPU vs GPU -- it takes another few seconds for detection to take place, at which time the rest of the lines with the results get printed.

The other thing is I don't believe you ran the command from the command-line yourself to test it. Without -dont_show darknet would normally pop up a OpenCV window to display the results. It doesn't exit until that window is dismissed.

Problem while reading stream from command line

1 Answers1