Python script : Running a script with multiple arguments using subprocess

Question

My question is related to this earlier question - Python subprocess usage

I am trying to run this command using python

nccopy -k 4 "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]" foo.nc

When I run the above command I should be able to see a file called foo.nc on my disk or a network error stating unable to access that URL or remote URL not found.

Currently the ESRL NOAA server is down - so when I run the above command I get

syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: ^ NetCDF: Access failure Location: file nccopy.c; line 1348

I should get the same error when I run the python script

This is the code I have and I am unable to figure out exactly how to proceed further -

I tried splitting up "-k 4" into two arguments and removing the quotes and I still get this error nccopy : invalid format : 4

Results of print(sys.argv) data.py

['data.py', '-k', '4', 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][20:34][26:40]', 'foo.nc']

import numpy as np

import subprocess

import sys

url = '"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]"'

outputFile = 'foo.nc'

arg1 = "-k 4"

arg3 = url 

arg4 = outputFile


print (input)

subprocess.check_call(["nccopy",arg1,arg3,arg4])

first of all, use [shlex](https://docs.python.org/3/library/shlex.html), secondly use `Popen()` instead of `check_call()` if possible. Also your problem is that you don't separate `-k` and `4`. Subprocess automatically tries to insert a space for each parameter element. Try this `["nccopy", '-k', '4']` — Torxed, May 30 '16 at 06:50
Also when calling system commands traditional shell logic still applies, meaning `" ` will enable parsing of variables and characters where as `' ` does not around parameters in the command. Unless you want parsing of the parameter I'd say use `' ` to wrap parameters. — Torxed, May 30 '16 at 06:53

score 2 · Answer 1 · edited May 23 '17 at 12:15

2

Instead of arg1 = "-k 4", use two arguments instead.

import subprocess


url = 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'

outputFile = 'foo.nc'

arg1 = "-k"
arg2 = "4"
arg3 = url 
arg4 = outputFile

subprocess.check_call(["nccopy", arg1, arg2, arg3, arg4])

See also here Python subprocess arguments

edited May 23 '17 at 12:15

Community

1
1

answered May 30 '16 at 06:51

SnoozeTime

363
1
9

Torxed · Accepted Answer · 2016-05-30T08:21:53.677

There's two dilemmas here.
One being that subprocess processes your arguments and tries to use 4 as a separate argument.

The other being that system calls still goes under normal shell rules, meaning that parameters and commands will be parsed for metacharacters aka special characters. In this case you're wrapping [ and ].

There for you need to separate each parameters and it's value into separate objects in the parameter-list, for instance -k 4 should be ['-k', '4'] and you need to wrap parameters/values in '...' instead of "...".

Try this, shlex.split() does the grunt work for you, and i swapped the encapsulation characters around the URL:

import numpy as np
import subprocess
import sys
import shlex

url = "'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'"

outputFile = 'foo.nc'
command_list = shlex.split('nccopy -k 4 ' + url + ' ' + outpufFile)

print(command_list)

subprocess.check_call(command_list)

If `outputFile` has a space in it; your code example fails. There is no reason to use `shlex.split()` here (and [it can be fooled in the general case](http://stackoverflow.com/questions/14386775/subprocess-call/14386815#comment45736245_14386815)). It is convenient to use `shlex.split()` in a REPL (to create a seed arguments list from a shell command, that you can fine-tune manually later)—avoid it in the final code. — jfs, May 30 '16 at 11:18
@J.F.Sebastian - can you write an alternate answer that is completely different to the other two ? — gansub, May 31 '16 at 03:04

score 1 · Answer 3 · edited May 23 '17 at 12:07

If you have a working shell command that runs a single program with multiple arguments and you want to parameterized it e.g., to use a variable filename instead of the hardcoded value then you could use shlex.split() to create a list of command-line arguments that you could pass to subprocess module and replace the desired argument with a variable e.g.:

>>> shell_command = "python -c 'import sys; print(sys.argv)' 1 't w o'"
>>> import shlex
>>> shlex.split(shell_command)
['python', '-c', 'import sys; print(sys.argv)', '1', 't w o']

To run the command using the same Python interpreter as the parent script, sys.executable could be used and we can pass a variable instead of '1':

#!/usr/bin/env python
import random
import sys
import subprocess

variable = random.choice('ab')
subprocess.check_call([sys.executable, '-c', 'import sys; print(sys.argv)',
                       variable, 't w o'])

Note:

one command-line argument per list item
no shlex.split() in the final code
there are no quotes inside 't w o' i.e., 't w o' is used instead of '"t w o"' or "'t w o'"

subprocess module does not run the shell by default and therefore you don't need to escape shell meta-characters such as a space inside the command-line arguments. And in reverse, if your command uses some shell functionality (e.g., file patterns) then either reimplement the corresponding features in Python (e.g., using glob module) or use shell=True and pass the command as a string as is. You might need pipes.quote(), to escape variable arguments in this case. Wildcard not working in subprocess call using shlex

Python script : Running a script with multiple arguments using subprocess

3 Answers3