Parsing non-mutually-exclusive groups of command-line arguments

Question

I am trying to find a way of parsing sequences of related arguments, preferably using argparse.

For example:

command --global-arg --subgroup1 --arg1 --arg2 --subgroup2 --arg1 --arg3 --subgroup3 --arg4 --subcommand1 --arg1 --arg3

where --global-arg applies to the whole command, but each --subgroupN argument has sub-arguments that apply only to it (and may have the same name, such as --arg1 and --arg3 above), and where some sub-arguments are optional, so the number of sub-arguments is not constant. However, I know that each --subgroupN sub-argument set is complete either by the presence of another --subgroupN or the end of the argument list (I am not fussed if global arguments cannot appear at the end, although I imagine that is possible as long as they don't clash with sub-argument names).

The --subgroupN elements are essentially sub-commands, but I do not appear to be able to use the sub-parser ability of argparse as it slurps any following --subgroupN entries as well (and therefore barfs with unexpected arguments).

(An example of this style of argument list is used by xmlstarlet)

Are there any suggestions beyond writing my own parser? I assume I can at least leverage something out of argparse if that is the only option...

Examples

The examples below were an attempt to find a way to parse an argument structure along the following lines:

(a --name <name>|b --name <name>)+

in the first example I hoped to have --a and --b introduce a set of arguments that were processed by a subparser.

I was hoping to get something out perhaps along the lines of

Namespace(a=Namespace(name="dummya"), b=Namespace(name="dummyb"))

subparser example fails

parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_a = subparsers.add_parser("a")
parser_b = subparsers.add_parser("b")
parser_a.add_argument("--name")
parser_b.add_argument("--name")
parser.parse_args(["a", "--name", "dummy"])
> Namespace(name='dummy') (Good)
parser.parse_args(["b", "--name", "dummyb", "a", "--name", "dummya"])
> error: unrecognized arguments: a (BAD)

mutually exclusive group fails

parser = argparse.ArgumentParser()
g = parser.add_mutually_exclusive_group()
g1 = g.add_mutually_exclusive_group()
g1.add_argument("--name")
g2 = g.add_mutually_exclusive_group()
g2.add_argument("--name")
> ArgumentError: argument --name: conflicting option string(s): --name (BAD)

(I wasn't really expecting this to work, it was an attempt to see if I could have repetition of grouped arguments.)

Please edit your question to show the code you tried. You'll get better answers that way. — msw, Jan 21 '16 at 11:15
There is at least one earlier SO question about handling multiple subcommands. Solutions involve things like recursive subcommands. None are straight forward uses of `argparse`. — hpaulj, Jan 21 '16 at 16:09
Searching for `[argparse] multiple` turned up posts like this, http://stackoverflow.com/q/25318622/901925 — hpaulj, Jan 21 '16 at 16:23

score 1 · Accepted Answer · edited May 23 '17 at 12:15

Other than the subparser mechanism, argparse is not designed to handle groups of arguments. Other than the nargs grouping, it handles the arguments in the order that they appear in the argv list.

As I mentioned in the comments there have been earlier questions, which can probably be found by search with words like multiple. But one way or other they seek to work about the basic order-independent design of argparse.

https://stackoverflow.com/search?q=user%3A901925+[argparse]+multiple

I think the most straight forward solution is to process the sys.argv list before hand, breaking it into groups, and then passing those sublists to one or more parsers.

parse [command --global-arg], 
parse [--subgroup1 --arg1 --arg2], 
parse [--subgroup2 --arg1 --arg3], 
parse [--subgroup3 --arg4], 
parse [--subcommand1 --arg1 --arg3]

In fact the only alternative is to use that subparser 'slurp everything else' behavior to get a remainder of arguments that can be parsed again. Use parse_known_args to return a list of unknown arguments (parse_args raises an error if that list is not empty).

OK - that sounds something along the lines of what I expected. I'll dig through the previous comments (hadn't found anything obvious when I searched earlier), but at least I know I was thinking along the right sort of lines! — MichaelNJ, Jan 21 '16 at 18:05

score 0 · Answer 2 · edited May 23 '17 at 11:52

Using hpaulj's reply above, I came up with the following:

args = [
    "--a", "--name", "dummya", 
    "--b", "--name", "dummyb",
    "--a", "--name", "another_a", "--opt"
]
parser_globals = argparse.ArgumentParser()
parser_globals.add_argument("--test")

parser_a = argparse.ArgumentParser()
parser_a.add_argument("--name")
parser_a.add_argument("--opt", action="store_true")

parser_b = argparse.ArgumentParser()
parser_b.add_argument("--name")

command_parsers = {
    "--a": parser_a,
    "--b": parser_b
}

the_namespace = argparse.Namespace()
if globals is not None:
    (the_namespace, rest) = parser_globals.parse_known_args(args)

subcommand_dict = vars(the_namespace)
subcommand = []
val = rest.pop()
while val:
    if val in command_parsers:
        the_args = command_parsers[val].parse_args(subcommand)
        if val in subcommand_dict:
            if "list" is not type(subcommand_dict[val]):
                subcommand_dict[val] = [subcommand_dict[val]]
            subcommand_dict[val].append(the_args)
        else:
            subcommand_dict[val] = the_args
        subcommand = []
    else:
        subcommand.insert(0, val)
    val = None if not rest else rest.pop()

I end up with:

Namespace(
    --a=[
        Namespace(
            name='another_a',
            opt=True
        ),
        Namespace(
            name='dummya',
            opt=False
        )
    ],
    --b=Namespace(
        name='dummyb'
    ),
    test=None
)

which seems to serve my purposes.

Parsing non-mutually-exclusive groups of command-line arguments

Examples

subparser example fails

mutually exclusive group fails

2 Answers2