4

I went through post after post on SO looking for a method to use quotation marks inside of arguments using subprocess.popen and I cannot seem to find a way.

This works fine from the commandline

runme.bat --include="check|check2"

Python

#!/usr/bin/python
import sys
import subprocess
import shlex

#command_line = "./runme.sh --include=\"check|check2\""
command_line = "runme.bat --include=\"check|check2\""

arg = shlex.shlex(command_line)
arg.quotes = '"'
arg.whitespace_split = True
arg.commenters = ''
command_line_args = list(arg)
print command_line_args

command_line_process = subprocess.Popen(
    command_line_args,
    universal_newlines=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

line = ""
while True:
    line = command_line_process.stdout.readline()
    if line:
        print line
        break

runme.bat

echo %* >> someargs.txt

runme.sh

#!/bin/bash
echo $@

I heard that subprocess.call() is a way around this but I'd like to be able to iterate line by line through the subprocess' output while the program is running.

Edit:

This seems to be a bug in Python because running runme.bat in cmd works correctly, running runme.py in linux works correctly, it's only when running runme.py on Windows where it doesn't work correctly. I created a ticket here.

Edit2:

It's not a python bug apparently lol. Look at chosen answer.

SomeGuyOnAComputer
  • 5,414
  • 6
  • 40
  • 72
  • The documentation is a bit unclear, but it looks as if using a string (rather than a sequence) for `args` might resolve the problem. Have you tried that already? If that doesn't work either, you might want to try filing *that* as a bug. Python should give you *some* way of passing an arbitrary command string without it trying to second-guess you. – Harry Johnston Apr 03 '15 at 23:23

3 Answers3

4

On Windows, a string is a native API. To avoid unnecessary conversions, pass the command as a string:

#!/usr/bin/env python
from __future__ import print_function
import subprocess

command = 'runme.bat --include="check|check2"'
process = subprocess.Popen(command,
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
    universal_newlines=True, bufsize=1)
for line in iter(process.stdout.readline, ''):
    print(line, end='')

stderr=subprocess.STDOUT merges stderr into stdout. If you set stderr=PIPE then you should read from process.stderr in parallel with reading from process.stdout otherwise your program may deadlock.

Popen() passes the string to CreateProcess() Windows function. If the child process is actually a batch-file; you should probably pass shell=True explicitly to make it clear that the command is interpreted using cmd.exe rules (^, |, etc are meta-characters, for more details read the links in this answer).

If you want to pass the argument using %1 instead of %* so that it includes
the whole --include="check|check2" (not only --include) then you could use additional quotes around the argument as @eryksun suggested in the comments:

command = '"runme.bat" "--include="check^^^|check2""'

Notice: triple ^ to escape | here.

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • (1) the command in the answer has no `shell=True` i.e., it is already assumes that `Popen()` (via `CreateProcess()`) can run `.bat` files. The command line is interpreted *differently* if it is eventually given to `cmd.exe` that is why I said to include `shell=True` for *documentation* purposes (it makes no difference otherwise as far as I can tell in this case). (2) I don't see any point to include quotes around `runme.bat` or to include `^^^` here (`^` is not special inside quotes). I'm sure there are many quirks accumulated over the years in batch-file syntax but the example works as is. – jfs Apr 06 '15 at 19:29
  • (1) "doesn't work" is not informative. I've used `py -c "import sys; print(sys.argv)" %1 %*` (in `runme.bat`) for testing and it works as expected. I'm sure; it may fail for other examples but I expect it to work for the case in the question. (2) It documents **the syntax** of the command line (`cmd.exe` rules) e.g., `shell=True` tells you whether `^` may be a meta-character. – jfs Apr 06 '15 at 22:14
  • @eryksun: I've included your more general command in the answer. Thank you for your comments. They are insightful as usual. – jfs Apr 07 '15 at 20:21
  • @Sebastian, brilliant! – SomeGuyOnAComputer Apr 10 '15 at 18:16
3

You should not use shell=True to run a bat file. Use it only if you have to run some built-in shell command. In other words the use your are making is useless and the only effect is to increase security vulnerability of your program.

Also, note that the documentation clearly states that, when using shell=True it's recommended to pass the command line as a string:

If shell is True, it is recommended to pass args as a string rather than as a sequence.

So you should do:

subprocess.check_output('runme.bat --include="check|check2"', shell=True)

The check_output function should be used if you only care for the output. It's way simper than creating a Popen object and then reading the output manually.

See also my answer regarding how shell=True changes the meaning of the arguments.

Community
  • 1
  • 1
Bakuriu
  • 98,325
  • 22
  • 197
  • 231
  • Thanks for the response. I just want to see the output line by line from the program while it's running which is possible with popen just not with arguments in double quotes. I removed the shell=True from the code above. You never mentioned the acknowledged the quotation issue in your answer... ? – SomeGuyOnAComputer Apr 03 '15 at 18:02
  • @Some note that using `shell=True` is also much slower than not using it, because it had to spawn a whole new shell and then execute the program. I don't get what your are saying about acknowledgements and I don't see any issue with that. I did disclose that I'm the author of the links answer. – Bakuriu Apr 04 '15 at 08:26
0

Another way to get the output is subprocess.check_output():

import subprocess

command_line = "runme.bat --include=\"check|check2\""
output = subprocess.check_output(
    command_line,
    shell=True
)
lines = output.splitlines(True)
print lines

To view the process's output in real time, see: Getting realtime output using subprocess.

Edit: here's code with Popen to handle double quotes:

from subprocess import Popen, PIPE, STDOUT

command_line = 'ls -la | grep "wheel"'
p = Popen(command_line, stdout=PIPE, stderr=STDOUT, shell=True)
while True:
    line = p.stdout.readline()
    if not line:
        break
    else:
        print line
Community
  • 1
  • 1
Joe Mornin
  • 8,766
  • 18
  • 57
  • 82
  • Thank you for your response but I'm looking for a way to run a command and capture output in realtime. I can do this with Popen but I also need to be able to use double quotes for a parameter... – SomeGuyOnAComputer Apr 03 '15 at 18:42
  • I've added a link to an answer about getting real time ouput. – Joe Mornin Apr 03 '15 at 18:44
  • The answer you linked uses Popen with real time output (which is what I'm using above) and does not use double quotes in their arguments. The answer you gave uses check_output with double quotes but doesn't output line by line in real time. Unless I misread the link? – SomeGuyOnAComputer Apr 03 '15 at 18:50
  • To handle the double quotes, you can wrap the command in single quotes, like this: `command_line = 'runme.bat --include="check|check2"'`. – Joe Mornin Apr 03 '15 at 18:52
  • I appreciate your suggestions but that does not work either. Please try the code I supplied above and you'll see why I posted about it. :) – SomeGuyOnAComputer Apr 03 '15 at 18:58
  • It seems that this works fine on Linux but it does not work fine on Windows. I also edited my first post and included a link to a bug that I just posted on the Python bugtracker. – SomeGuyOnAComputer Apr 03 '15 at 19:23
  • OK—curious to see how it turns out. Feel free to accept the answer if it was helpful. – Joe Mornin Apr 03 '15 at 19:29
  • I'm sorry but as hard as you tried you weren't that helpful... Thank you for attempting to answer my question. – SomeGuyOnAComputer Apr 03 '15 at 21:16