Python Popen - Unknown argument: |

Question

I am getting Unknown argument: | when using the below script and I'm not sure why it doesn't like the |. I am on Windows, with Python 2.7.11:

import subprocess
from subprocess import Popen, PIPE, STDOUT
import time

command = 'VSPipe.exe --y4m script.vpy - | ffmpeg.exe -f yuv4mpegpipe -i - -c:v prores -an output.mov'

process1 = Popen(command, stderr=PIPE, shell=False)

while True:
    line = process1.stderr.readline().decode('utf-8')
    print line
    time.sleep(2)

UPDATE: Using the suggestion below I have now tried this, but I am getting pipe:: Operation not permitted. What am I doing wrong?

import subprocess
from subprocess import Popen, PIPE, STDOUT
import time

command1 = 'VSPipe.exe --y4m script.vpy -'
command2 = 'ffmpeg.exe -f yuv4mpegpipe -i - -c:v prores -an output.mov'

process1 = Popen(command1, stdout=PIPE)
process2 = Popen(command2, stdin=process1.stdout, stderr=PIPE)

print process2.communicate()

ANOTHER UPDATE: So using the suggestions below, and trying other stuff, I keep getting the same result pipe:: Operation not permitted. Here is the full sdterr, what else can I try:

"F:/ffmpeg/VapourSynth/VSPipe.exe" --y4m "C:/Users/myself/Desktop/script.vpy" - 
"F:/ffmpeg/ffmpeg.exe" -f yuv4mpegpipe -i - -c:v prores -an -y "//192.168.0.100/media/temp/OutMov.mov"
ffmpeg version N-77203-gb8e5b1d Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --arch=x86 --target-os=mingw32 --cross-prefix=/Users/myself/Desktop/Download/ffmpeg-windows-build-helpers-master/sandbox/mingw-w64-i686/bin/i686-w64-mingw32- --pkg-config=pkg-config --disable-w32threads --enable-gpl --enable-libsoxr --enable-fontconfig --enable-libass --enable-libutvideo --enable-libbluray --enable-iconv --enable-libtwolame --extra-cflags=-DLIBTWOLAME_STATIC --enable-libzvbi --enable-libcaca --enable-libmodplug --extra-libs=-lstdc++ --extra-libs=-lpng --enable-libvidstab --enable-libx265 --enable-decklink --extra-libs=-loleaut32 --enable-libx264 --enable-libxvid --enable-libmp3lame --enable-version3 --enable-zlib --enable-librtmp --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libopenjpeg --enable-gnutls --enable-libgsm --enable-libfreetype --enable-libopus --enable-frei0r --enable-filter=frei0r --enable-libvo-aacenc --enable-bzlib --enable-libxavs --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libschroedinger --enable-libvpx --enable-libilbc --enable-libwavpack --enable-libwebp --enable-libgme --enable-dxva2 --enable-libdcadec --enable-avisynth --enable-gray --enable-libopenh264 --extra-libs=-lpsapi --extra-cflags= --enable-static --disable-shared --prefix=/Users/myself/Desktop/Download/ffmpeg-windows-build-helpers-master/sandbox/mingw-w64-i686/i686-w64-mingw32 --enable-nonfree --enable-libfdk-aac --disable-libfaac --enable-nvenc --enable-runtime-cpudetect
  libavutil      55. 10.100 / 55. 10.100
  libavcodec     57. 17.100 / 57. 17.100
  libavformat    57. 19.100 / 57. 19.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 20.100 /  6. 20.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
pipe:: Operation not permitted

UPDATE: And her is the ffmpeg report:

F:/ffmpeg/ffmpeg.exe -f yuv4mpegpipe -i - -c:v prores -an -y //192.168.0.100/media/temp/OutMov.mov -report
ffmpeg version N-77203-gb8e5b1d Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --arch=x86 --target-os=mingw32 --cross-prefix=/Users/myself/Desktop/Download/ffmpeg-windows-build-helpers-master/sandbox/mingw-w64-i686/bin/i686-w64-mingw32- --pkg-config=pkg-config --disable-w32threads --enable-gpl --enable-libsoxr --enable-fontconfig --enable-libass --enable-libutvideo --enable-libbluray --enable-iconv --enable-libtwolame --extra-cflags=-DLIBTWOLAME_STATIC --enable-libzvbi --enable-libcaca --enable-libmodplug --extra-libs=-lstdc++ --extra-libs=-lpng --enable-libvidstab --enable-libx265 --enable-decklink --extra-libs=-loleaut32 --enable-libx264 --enable-libxvid --enable-libmp3lame --enable-version3 --enable-zlib --enable-librtmp --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libopenjpeg --enable-gnutls --enable-libgsm --enable-libfreetype --enable-libopus --enable-frei0r --enable-filter=frei0r --enable-libvo-aacenc --enable-bzlib --enable-libxavs --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libschroedinger --enable-li  libavutil      55. 10.100 / 55. 10.100
  libavcodec     57. 17.100 / 57. 17.100
  libavformat    57. 19.100 / 57. 19.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 20.100 /  6. 20.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Splitting the commandline.
Reading option '-f' ... matched as option 'f' (force format) with argument 'yuv4mpegpipe'.
Reading option '-i' ... matched as input file with argument '-'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'prores'.
Reading option '-an' ... matched as option 'an' (disable audio) with argument '1'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '//192.168.0.100/media/temp/OutMov.mov' ... matched as output file.
Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option report (generate a report) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input file -.
Applying option f (force format) with argument yuv4mpegpipe.
Successfully parsed a group of options.
Opening an input file: -.
[AVIOContext @ 006d9800] Statistics: 0 bytes read, 0 seeks
pipe:: Operation not permitted

UPDATE: Using this code, I still get the same result pipe:: Operation not permitted:

import subprocess
import os
from subprocess import Popen, PIPE, STDOUT
import shlex

VsPipe = 'F:/ffmpeg/VapourSynth/VSPipe.exe'
vpyScript = 'C:/Users/myself/Desktop/Boychoir-preview_SMALL_JobID_189.vpy'
ffmpeg = 'F:/ffmpeg/ffmpeg.exe'
outputPath = '//192.168.0.100/media/temp/OutMov.mov'

command1 = shlex.split('"%s" --y4m "%s" - ' % (VsPipe, vpyScript))
command2 = shlex.split('"%s" -f yuv4mpegpipe -i - -c:v prores -an -y "%s" -report' % (ffmpeg, outputPath))

process1 = subprocess.Popen(command1, stdout=subprocess.PIPE)
ls_out, _ = process1.communicate()
process2 = subprocess.Popen(command2, stdin=subprocess.PIPE, stderr=subprocess.PIPE)

grep_out, grep_err = process2.communicate(input=ls_out)
print grep_err

UPDATE: Fixed, turned out to be a file path permissions issue, thanks for all the help.

You can't use '|' in command to pipe command together refer to http://stackoverflow.com/questions/7389662/link-several-popen-commands-with-pipes. — Ali SAID OMAR, Jan 20 '16 at 09:32
Found this http://stackoverflow.com/questions/9356960/ffmpeg-operation-not-permitted-error-while-conversion — Ali SAID OMAR, Jan 20 '16 at 14:10

score 2 · Accepted Answer · answered Jan 20 '16 at 09:51

When you set shell to False the args (ie the first) argument is supposed to be the name of a program if it's a string. Otherwise it's supposed to be a a sequence of strings, the first being the program and the rest being the arguments.

The importance of a shell when using pipes is that in the line:

VSPipe.exe --y4m script.vpy - | ffmpeg.exe -f yuv4mpegpipe -i - -c:v prores -an output.mov

all is not considered arguments. The arguments to VSPipe.exe are --y4m and script.vpy. The arguments to ffmpeg.exe are -f, yuv4mpegpipe, -i etc. The shell is whats handling the pipe - it starts both programs (with their arguments supplied), but it does so by first creating a pipe so that stdout from the first becomes connected to stdin of the second.

If you don't have a shell that supports piping you could emulate that in python by starting them both and then transport output from the first to the second.

afunkyrobot · Answer 2 · 2016-01-20T11:48:52.003

From the subprocess docs:

Use Popen with the communicate() method when you need pipes.

Building on @skyking's answer and your updated code, it looks like you just need to use communicate():

# Simple Windows script to confirm existence of Desktop directory in users home space
# ls -al | grep Desktop
# This requires Windows Git to be installed for unix utils like ls, grep

# NOTE: you may have problems with this if you're using built in Windows shell
# utilities like dir etc.. This is because these are built into the shell and
# aren't executables on your path. Use shell=True if you have to use these.

import os
import subprocess

# Change to home directory
os.chdir(os.path.expanduser("~"))

# Command parameters sent to Popen initializer should be lists
# Here we are only interested in the output so PIPE stdout
ls_process = subprocess.Popen(['ls', '-al'], stdout=subprocess.PIPE)

# When working with subprocess PIPE we need to use the communicate() method
# We're only interested in stdout so _ is used to throw away stderr (which will be
# None anyway since we didn't pipe stderr)
ls_out, _ = ls_process.communicate()

# We then create our second process but this time we're interested in piping the
# first command into stdin, using stdout and potentially stderr if there was a problem
grep_process = subprocess.Popen(['grep', 'Desktop'],
                                   stdin=subprocess.PIPE,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE)

# We use communicate() again but this time we give the output string of the first command
# as a parameter to communicate()
grep_out, grep_err = grep_process.communicate(input=ls_out)

print grep_out

Subprocess is a difficult module to get right first time and it's great to look at solid examples of it in use in other projects.

Another good tip is to try and keep things as platform independent as possible. Even if you have no intention to run this on anything other than Windows it will help you avoid the idiosyncrasies of the system you're working on.

I have amended the question to include `print process2.communicate()` but I still get `pipe:: Operation not permitted` — speedyrazor, Jan 20 '16 at 11:04
I have updated my code to show use of the `communicate()` method. Hope this helps! — afunkyrobot, Jan 20 '16 at 11:49
Sorry, no, same result, I have updated my question with ANOTHER UPDATE: — speedyrazor, Jan 20 '16 at 12:29
@speedyrazor In the code you have in your updated question, you don't pipe `stdout` in the second process. Because of this `process2` executes as soon as `process2 = Popen(command2, stdin=process1.stdout, stdout=PIPE, stderr=PIPE)` is evaluated. You then call `process2.communicate()` which tries executing it again but with an empty input buffer (as it has already been consumed). When you use pipes and `communicate()` you ALWAYS have to pipe `stdout`. — afunkyrobot, Jan 20 '16 at 12:42
OK, updated the question again, incorporating your suggestion, same result pipe:: Operation not permitted. — speedyrazor, Jan 20 '16 at 13:01

score 0 · Answer 3 · answered Jan 20 '16 at 11:19

First I would recommand you to user full path for all .exe here a code I used to use to do that.

import os
import re
import subprocess
from subprocess import Popen, PIPE, STDOUT
import shlex

def _get_locate_command_result(bin_name,
                               which_command="which", default_path=""):
    # check_output returns bytes for python3 and str for py2
    try:
        result_locate = subprocess.check_output(
            shlex.split("%s %s" % (which_command, bin_name)), env=os.environ)
        result_locate_table = re.split(r"\r?\n", result_locate.decode())
        try:
            bin_path = result_locate_table[0].strip()
        except IndexError:
            bin_path = bin_name
        return bin_path


    except subprocess.CalledProcessError:
        return default_path

For Windows:

path = get_locate_command_result(bin_name, which_command="where", default_path=default_path)

So you could try something like that (not tested):

# use "" beacause of '\'
command1 = ('"%s" --y4m script.vpy -'
            % _get_locate_command_result("VSPipe.exe", "where", "VSPipe.exe"))
command2 = ('"%s" -f yuv4mpegpipe -i - -c:v prores -an output.mov'
            % _get_locate_command_result("VSPipe.exe", "where", "ffmpeg.exe"))

# Popen needs sequence of string
cmd1 = shlex.split(command1)
cmd2 = shlex.split(command2)

process1 = Popen(cmd1, stdout=PIPE)
process2 = Popen(cmd2, stdin=process1.stdout, stderr=PIPE)

print(process2.communicate())

I am already using full path for all .exe files in my code, I excelued it here for sake of brevity. Assume the script can find all exe's and files. Also this is for Windows only. — speedyrazor, Jan 20 '16 at 11:42
I should use shlex.split(cmd) to return a sequence of string. — Ali SAID OMAR, Jan 20 '16 at 13:08
Added in shlex.split(cmd), but still go the same pipe:: Operation not permitted — speedyrazor, Jan 20 '16 at 13:33

Python Popen - Unknown argument: |

3 Answers3