How to automate running python code with every combination of command line options

Question

I have written some python code that takes command line options. I'd like to automate running every combination, instead of having to manually type all combinations.

At the moment I have a perl script that looks something like this (although with more options an therefore more nested foreach loops):

use Cwd;
$curdir=getcwd;
@opt1 = ("", "-s");
@opt2 = ("","-w \'binary\'", "-w \'tf\'", "-w \'tfidf\'") #There are arguments but there is a finite amount of them.

foreach $s (@opt1) {
    foreach $t (@opt2) {
        $cmd="python myCode.py $s $t";
        system($cmd);
    }
}

This results in the following being run python myCode.py -t 'binary',python myCode.py -t 'tf',python myCode.py -t 'tfidf',python myCode.py -s -t 'binary',python myCode.py -s -t 'tf',python myCode.py -s -t 'tfidf'.

This seems like a horrible way to do it (considering I actually have many more options), is there a proper or better way? I'm open to using python to automate this, I have used perl initially as I have a perl script goes onto to call other programs.

Are you unit testing? If so, [this question](http://stackoverflow.com/questions/32899/how-to-generate-dynamic-parametrized-unit-tests-in-python) may help. Or do you really just want to know how to get that specific pattern of command line flags? — KevinOrr, Mar 21 '15 at 23:06

jedwards · Answer 1 · 2015-03-21T23:27:50.507

0

You could do this inside your python file. Wrap everything in a function, then handle the combinations of command line arguments like:

import sys
import itertools

args = sys.argv[1:]

for i in range(len(args)+1):
    for c in itertools.combinations(args, i):
        print c  # You'd put your function call here

With mycode.py -a -b -c, the iterations of c are:

()
('-a',)
('-b',)
('-c',)
('-a', '-b')
('-a', '-c')
('-b', '-c')
('-a', '-b', '-c')

And with mycode.py -a -b -c -d, the iterations of c are:

()
('-a',)
('-b',)
('-c',)
('-d',)
('-a', '-b')
('-a', '-c')
('-a', '-d')
('-b', '-c')
('-b', '-d')
('-c', '-d')
('-a', '-b', '-c')
('-a', '-b', '-d')
('-a', '-c', '-d')
('-b', '-c', '-d')
('-a', '-b', '-c', '-d')

So your code might then look like:

import sys
import itertools

def your_function(args):
    print("your_function called with args=%s" % str(args))
    '''
    SOME LONG FUNCTION HERE
    '''


# Call your_function on all r-length combinations of command line arguments
#   where 0 <= r <= len(args)
if __name__ == "__main__":
    args = sys.argv[1:]

    for i in range(len(args)+1):
        for c in itertools.combinations(args, i):
            your_function(c)

edited Mar 21 '15 at 23:27

answered Mar 21 '15 at 23:08

jedwards

29,432
3
65
92

This is a great answer, but what about if the options are not just true/false but also take an input, like the commented out lines in the example? – jksnw Mar 21 '15 at 23:15
@jksnw: You haven't said what you want to do in that case, so we can't really help you. If you add an example of such an option to your question and show the sort of result you require, then we will be able to suggest something. – Borodin Mar 21 '15 at 23:19
@jksnw well, you could wrap the option and value in quotes, e.g. `mycode.py "-a arg1" "-b arg2" "-c arg3"`, then they'd be preserved as strings through to your function -- after that, it would depend on how you were handling command line arguments in the first place (in your code currently) – jedwards Mar 21 '15 at 23:22
Some arguments have an input but there are a finite number of inputs, the code has some if statements such as `if opts['-w'] == 'binary': run some code`. – jksnw Mar 22 '15 at 00:00
@jksnw what method are you using to go from `sys.argv` to your `opts` dictionary? – jedwards Mar 22 '15 at 01:26
`opts, args = getopt.getopt(sys.argv[1:], 'hrsw:ctpm:l:')` `opts = dict(opts)` – jksnw Mar 22 '15 at 12:38
@jksnw do any of your options include spaces and require you to wrap them in double quotes? If not, there is an easy solution, e.g. something like `sys.argv = [sys.argv[0]] + " ".join(c).split()` -- then your `getopt` stuff can remain unchanged. – jedwards Mar 23 '15 at 13:21

score 0 · Answer 2 · answered Mar 21 '15 at 23:13

I suggest that you use the combinations function from the Perl Algorithm::Combinatorics module on CPAN. It isn't a core module and so will probably need to be installed.

use strict;
use warnings;

use Algorithm::Combinatorics qw/ combinations /;

my @switches = qw/ -a -b -c /;

for my $n ( 0 .. @switches ) {
  for my $combination ( combinations(\@switches, $n) ) {
    print join(' ', qw/ python myCode.py /, @$combination), "\n";
  }
}

output

python myCode.py
python myCode.py -a
python myCode.py -b
python myCode.py -c
python myCode.py -a -b
python myCode.py -a -c
python myCode.py -b -c
python myCode.py -a -b -c

jksnw · Answer 3 · 2015-03-23T17:07:09.137

This is what I ended up doing:

use Math::Cartesian::Product;
cartesian {system("python myCode.py @_")} ["","-s"], ["-w \'tf\'", "-w \'binary\'", "-w \'tfidf\'"];

In python itertools.product does what I was looking for:

import itertools

opts_list = [["","s"], ["tf","binary","tfidf"]]

print list(itertools.product(*opts_list))

Prints:

[('', 'tf'), ('', 'binary'), ('', 'tfidf'), ('s', 'tf'), ('s', 'binary'), ('s', 'tfidf')]

I changed my code slightly to use this method but I suppose each element in the tuple could be concatenated into a string in order to run the options in command line.

score -1 · Answer 4 · edited May 23 '17 at 12:28

-1

This answer about cartesian products should solve the problem for you in Python.

edited May 23 '17 at 12:28

Community

1
1

answered Mar 21 '15 at 23:08

Sigfried

2,943
3
31
43

The link has a good answer, it uses python and doesn't use a load of for loops like my example would do if there were more options. It could also be used if there were arguments in the options. – jksnw Mar 21 '15 at 23:42

How to automate running python code with every combination of command line options

4 Answers4