Passing list as an argument in python script in a subproces call

Question

I am using Subprocess call to execute Insert_API from main script.Values to be inserted are stored in a list and I need to pass the list as an argument while subprocess call. It gives me error saying arguments must be strings . If I make str(list) then it works but Insert does not work properly as its passed as string not as list. Tried with passing as string and converting again to list in script but that also does not work properly. Is there any way to pass List as it is ? I have 2 lists to pass everytime. Kindly, Let me know if there is any way to pass 2 lists in subprocess call ? Sample code :

subprocess.check_output(["python","Metadata_Index_EAV_LOAD.py",Dataset1,MetaDataIndex_Table,SchemaFields,FieldValues,r_date])

SchemaField and result are two lists that I need to pass as an argument.

Answer 1

You could set up argparse in your mainScript to handle input arguments using

parser = argparse.ArgumentParser()
parser.add_argument('-sf','--schema_field', action='append', help="List of id's")
args = parser.parse_args()

this way in your command line argument you just add id's to the list by re-using the shortcut.

python -c mainScript.py -sf 1001 -sf 1002 -sf 1003

And this should return args.schema_field as a list inside your mainScript.

Handling complex optional and positional commandline arguments is definitely the job for argparse - so I recommend going over the official docs here : https://docs.python.org/2/howto/argparse.html

Answer 2

You probably could find a way of passing the arguments as list such as suggested in this answer but still maybe an easier solution would be to use the json.loads method.

Instead of running list(SchemaFields) for instance try running something like:

import json
schema_list = json.loads(SchemaFields)
values = json.loads(FieldValues)

And you probably will have to keep sending the arguments as strings.

Still, I agree with Roseman's comment, it just seems if you used some base class with all these operations ready to go and just invoked them from your main script this would be much simpler (not sure what you mean by hard-coding values this way as still you could send sys.args input in the main script).

Answer 3

Perhaps you wish to pass a list through args, but this shouldn't be done so. Why? Because length of the command line passed to the kernel is limited. Newest Linux distros accept pretty long strings, but still. When you have a lot of stuff that can contain special characters and so on, it sometimes may go terribly wrong. Especially if you will give usage of the receiver script to other people, who can forget to bypass shell and send the data directly to your script. And in this case whole mess can rise.

So, I advise you to use pipes. I.e. in receiver read from sys.stdin, and in sender use Popen().communicate(). Something like this:

# Receiver:
from cPickle import loads
import sys

data = sys.stdin.read() # This will return after '^D' is received.
data = loads(data)
print "I received:"
for x in data:
    print x
    print "\t", data[x]

# Sender:
from cPickle import dumps
from subprocess import Popen, PIPE

data = {"list1": [1, 2, 3, 4],
    "list2": [3, 4, 5, 6]}
cmd = ["python", "receiver.py"]
p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
result, error = p.communicate(dumps(data))
if error:
    raise Exception(error)
print "Receiver sent:"
print result

Now, if other people need not to use Python to send data to your script, then do not use pickling but json as suggested in another answer. Or you can even use literal_eval():

# In receiver:
from ast import literal_eval
import sys

data = sys.stdin.read()
data = literal_eval(data)
#...

# And in sender:
# ...
result, error = p.communicate(repr(data))
#...

literal_eval() is a secure version of eval() that will evaluate only built-in data types, not expressions and other objects.

If you still insist on using args, then you should use something like:

list1 = [1, 2, 3, 4]

def convert (l):
    return ",".join((str(x) for x in l))

def get (s):
    return s.split(",")

But then you have to make sure that your separator isn't appearing anywhere else in the string or make an escaping mechanism. Also if you would not be sending only alpha-numeric characters you would have to encode the stuff using URL-safe base64 to avoid possible clashes with OS and/or shell.