passing special chars as args from python script to shell

Question

I have a config_file.yml file like this:

sample:
    sql: "select * from dbname.tableName where sampleDate>='2018-07-20';"
    config: {'hosts': [!!python/tuple ['192.162.0.10', 3001]]}

sample2:
    sql: "select * from dbname.tableName where sampleDate<='2016-05-25';"
    config: {'hosts': [!!python/tuple ['190.160.0.10', 3002]]}

My python code is:

data_config = yaml.load(config_file)
for dataset, config in data_config.items():
    args = [config]
    cmd = ['./execute_something.sh']
    cmd.extend(args)
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True).communicate()

execute_something.sh:

#!/bin/bash
echo $1
data_config=$1
echo $data_config

So basically I want to pass {'sql': "select * from dbname.tableName where sampleDate>='2018-07-20';", config: {'hosts': [!!python/tuple ['190.160.0.10', 3002]]}} this entire string as an argument to the shell script.

Problem: 1) select * ends up listing all files in the current directory, instead of being passed entirely as a string 2) even if I pass a simple string say args="hi" it still won't work!

I don't understand what I am missing here. Kindly help. Thanks!

It's nothing to do with special characters -- you'd have the same problem with this code passing arguments with no special characters at all. — Charles Duffy, Jul 20 '18 at 16:23
BTW, you should also be careful about quoting on the shell end of the world -- `echo "$1"` is more correct than `echo $1`. See [BashPitfalls #14](http://mywiki.wooledge.org/BashPitfalls#echo_.24foo). — Charles Duffy, Jul 20 '18 at 16:26

score 1 · Accepted Answer · answered Jul 20 '18 at 16:19

DO NOT USE `shell=True`.

data_config = yaml.load(config_file)
for dataset, config in data_config.items():
    cmd = ['./execute_something.sh', str(config)]
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()

When you run shell=True, you're prepending sh -c to your literal argument list. What that does in this case is the following (escaping is added to make the single quotes literal):

sh -c './execute_something.sh' '{'"'"'sql'"'"': "select * from dbname.tableName where sampleDate>='"'"'2018-07-20'"'"';", config: {'"'"'hosts'"'"': [!!python/tuple ['"'"'190.160.0.10'"'"', 3002]]}}'

That doesn't work. Try it manually in a shell, if you like. Why? Because the argument starting with the { isn't passed to ./execute_something.sh, but is instead passed to the shell executing sh -c.

What would work, if you really insisted on keeping shell=True? Compare to the following:

sh -c './execute_something.sh "$@"' _ '{'"'"'sql'"'"': "select * from dbname.tableName where sampleDate>='"'"'2018-07-20'"'"';", config: {'"'"'hosts'"'"': [!!python/tuple ['"'"'190.160.0.10'"'"', 3002]]}}'

Here, the argument just after the -c is a shell script that looks at its arguments, and passes those arguments on to execute_something.sh.

passing special chars as args from python script to shell

1 Answers1

DO NOT USE shell=True.

DO NOT USE `shell=True`.