7

I am writing a wrapper to automate some android ADB shell commands via Python (2.7.2). Since, in some cases, I need to run the command asynchronously, I am using the subprocess.Popen method to issue shell commands.

I have run into a problem with the formatting of the [command, args] parameter of the Popen method, where there require command/args split is different between Windows and Linux:

# sample command with parameters
cmd = 'adb -s <serialnumber> shell ls /system'

# Windows:
s = subprocess.Popen(cmd.split(), shell=False) # command is split into args by spaces

# Linux:
s = subprocess.Popen([cmd], shell=False) # command is a list of length 1 containing whole command as single string

I have tried using shlex.split(), with and with out the posix flag:

# Windows
posix = False
print shlex.split(cmd, posix = posix), posix
# Linux
posix = True
print shlex.split(cmd, posix = posix), posix

Both cases return the same split.

Is there a method in subprocess or shlex that handles the OS-specific formats correctly?

This is my current solution:

import os
import tempfile
import subprocess
import shlex

# determine OS type
posix = False
if os.name == 'posix':
    posix = True

cmd = 'adb -s <serialnumber> shell ls /system'
if posix: # posix case, single command string including arguments
    args = [cmd]
else: # windows case, split arguments by spaces
    args = shlex.split(cmd)

# capture output to a temp file
o = tempfile.TemporaryFile()
s = subprocess.Popen(args, shell=False, stdout=o, stderr=o)
s.communicate()
o.seek(0)
o.read()
o.close()

I don't think shlex.split() is doing anything here, and cmd.split() achieves identical results.

Nisan.H
  • 6,032
  • 2
  • 26
  • 26
  • You made a typo in the question. shlex vs shelx. – jgritty Feb 17 '12 at 02:36
  • @J.F.Sebastian a previous version of the (full) code actually failed with out it. I have changed it `shell=False` now. Thanks for the remark. – Nisan.H Feb 17 '12 at 02:53
  • unrelated: `s.communicate()` is a blocking call it waits the process to terminate (it is not asynchronous) so you could use `output = subprocess.check_output(args, stderr=STDOUT)` instead of `output = o.read()`. – jfs Feb 17 '12 at 03:11
  • @J.F.Sebastian I know. The sample command here is not continuous, and having s.communicate block until the process terminates makes sense. An asynchronous case would be capturing the output of `getevent` or `logcat` up until a termination command is issued from the PC client. (Strictly speaking, you don't need to do this for logcat, as you can just read the logfiles, but you do for getevent.) – Nisan.H Feb 17 '12 at 12:10
  • `s.split()` and `shlex.split(s)` are not identical! Try comparing, for example, the following in the Python console: `shlex.split('echo "foo bar is fun $(ls /tmp)"', posix=False)` and `'echo "foo bar is fun $(ls /tmp)"'.split()`. The classic `str.split()` method splits a string logically, but the `shlex.split(str)` module splits a string intelligently - as understood by Shell. – s3n0 Nov 01 '19 at 10:54

2 Answers2

7

They seem to function identically when I turn off shell=True

As per the docs:

On Unix, with shell=True: If args is a string, it specifies the command string to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

http://docs.python.org/library/subprocess.html

jgritty
  • 11,660
  • 3
  • 38
  • 60
  • 1
    The shell taking the arguments, rather than the command executed by it, explains the problem (and the solution) perfectly. With `shell=False` `args = cmd.split()` and `subprocess.Popen(args, shell=False)` behave identically on both linux and windows. – Nisan.H Feb 17 '12 at 03:02
5

The shell=True argument tells it to have the command line evaluated by your shell, which on Windows will be Cmd.exe; on Linux, it'll likely be /bin/bash, but could also be some other related shell (zsh, tcsh, etc.). The difference in behavior is likely being caused by the shells interpreting the commands differently.

I'd strongly recommend not using shell=True if you can avoid it. Just something like this:

cmd = 'adb -s <serialnumber> shell ls /system'
s = subprocess.Popen(cmd.split())  # shell=False by default
Adam Rosenfield
  • 390,455
  • 97
  • 512
  • 589
  • Unfortunately, cmd.split() will fail in some cases, ie when an argument has spaces, like `cmd='/usr/bin/ls "/home/user/my directory"'` will fail with cmd.split(). In that case, `cmd = shlex.split(cmd, posix=True)` will work better. But shlex.split() will fail when stdout is captured, hence there is no allrounder solution IMHO. – Orsiris de Jong Dec 16 '20 at 10:20