5

I'm using argparse with several subparsers. I want my program to take options for verbosity anywhere in the args, including the subparser.

from argparse import ArgumentParser
p = ArgumentParser()
p.add_argument('--verbose', '-v', action='count')

sub = p.add_subparsers()
a = sub.add_parser('a')

print(p.parse_args())

By default, options for the main parser will throw an error if used for subparsers:

$ python tmp.py -v a
Namespace(verbose=1)

$ python tmp.py a -v
usage: tmp.py [-h] [--verbose] {a} ...
tmp.py: error: unrecognized arguments: -v

I looked into parent parsers, from this answer.

from argparse import ArgumentParser

parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')

main = ArgumentParser(parents=[parent])

sub = main.add_subparsers()
a = sub.add_parser('a', parents=[parent])

print(main.parse_args())

For some reason though, none of the shared flags work on the main parser.

$ python tmp2.py a -vvv
Namespace(verbose=3)
$ python tmp2.py -vvv a
Namespace(verbose=None)

Note that the main parser definitely has the appropriate arguments, because when I change it to main = ArgumentParser() I get error: unrecognized arguments: -v. What am I missing here?

jyn
  • 463
  • 4
  • 16
  • 2
    You could add `argument_default=SUPPRESS` to the parent parser, but that just prevents the subparser from applying its default if the subparser doesn't use `-v` at all. Otherwise, the subparser still overrides anything does by the main parser, rather than augmenting it. – chepner May 26 '18 at 15:46
  • I think the answer is to define a `SharedCountAction` rather than using the builtin `_CountAction` (specified by `action='count')`, but I am not sure what that definition would look like. – chepner May 26 '18 at 15:47
  • 1
    I looked at [`_CountAction`](https://github.com/python/cpython/blob/master/Lib/argparse.py#L995) and it look about right. I think the problem is in [`_SubParsersAction.__call__`](https://github.com/python/cpython/blob/master/Lib/argparse.py#L1149), which overwrites any existing attributes with that of the subparser. – jyn May 26 '18 at 16:06
  • this seems to support that: if the script is called without subparsers, it works fine: `$ python tmp2.py -v` gives `Namespace(verbose=1)` – jyn May 26 '18 at 16:16

2 Answers2

3

First, a couple of general comments.

The main parser handles the input upto the subparser invocation, then the subparser is called and given the remaining argv. When it is done, it's namespace is merged back into the the main namespace.

The parents mechanism copies Actions from the parent by reference. So your main and subparsers share the same verbose Action object. That's been a problem when the subparser tries to set a different default or help. It may not be an issue here, but just keep it in mind.

Even without the parents mechanism, sharing a dest or options flag between main and subparser can be tricky. Should the default of the subparser Action be used? What if both are used? Does the subparser overwrite the main parser's actions?

Originally the main namespace was passed to the subparser, which it modified and returned. This was changed a while back (I can find the bug/issue if needed). Now the subparser starts with a default empty namespace, fills it. And these values are then merged into the main.

So in your linked SO question, be wary of older answers. argparse may have changed since then.

I think what's happening in your case is that the main and subparser verbose are counting separately. And when you get None it's the subparser's default that you see.

The __call__ for _Count_Action is:

def __call__(self, parser, namespace, values, option_string=None):
    new_count = _ensure_value(namespace, self.dest, 0) + 1
    setattr(namespace, self.dest, new_count)

I suspect that in older argparse when the namespace was shared, the count would have been cumulative, but I can't test it without recreating an older style subparser action class.

https://bugs.python.org/issue15327 - here the original developer suggests giving the two arguments different dest. That records the inputs from both main and sub. Your own code can then merge the results if needed.

https://bugs.python.org/issue27859 argparse - subparsers does not retain namespace. Here I suggest a way of recreating the older style.

https://bugs.python.org/issue9351 argparse set_defaults on subcommands should override top level set_defaults - this is the issue in 2014 that changed the namespace use.

hpaulj
  • 221,503
  • 14
  • 230
  • 353
  • This is interesting, but it doesn't answer my question. You mentioned that `your main and subparsers share the same verbose Action object`. This is good, that's what I want to happen and what I would expect. Since both of these use 'count', I would expect the effect to be additive, i.e. `myprog -v a -v` should have a verbose count of 2. – jyn May 26 '18 at 16:00
  • Actually the `count` is done in the `namespace`. It is done by the `Action`, but it doesn't modify its own values. I added the `__call__` code. – hpaulj May 26 '18 at 16:05
  • 1
    `argparse` has been pretty stable, but it probably wasn't designed to handle this use case. – chepner May 26 '18 at 16:08
  • I wrote this above - I think the problem is in [`_Subparser.__call__`](https://github.com/python/cpython/blob/master/Lib/argparse.py#L1149), the parent value is being overridden by the subparser value. Would it be appropriate to file a bug report for this? Or is this the expected behaviour? – jyn May 26 '18 at 16:12
  • ~~actually while I'm at it, I have python2.7 installed, let me see if the behaviour is the same~~ behavior is the same in 2.7.14 as 3.6.2 – jyn May 26 '18 at 16:14
  • I like your idea of different `dest` for each and then adding. I think that's what I'll do. – jyn May 26 '18 at 16:21
0

My workaround for this behavior, which is very well described in @hpaulj's answer is to create a second parser that does not have subparsers but only the positional arguments that were first found.

The first parse_args, used with the first parser, will validate the positional arguments and flags, show an error message if needed or show the proper help.

The second parse_args, for the second parser, will correctly fill in the namespace.

Building on your example:

from argparse import ArgumentParser

parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')

main1 = ArgumentParser(parents=[parent])
sub = main1.add_subparsers()

# eg: tmp.py -vv a -v
a = sub.add_parser('a', parents=[parent])
a.set_defaults(which='a')

# eg: tmp.py -vv v -v --output toto
b = sub.add_parser('b', parents=[parent])
b.add_argument('--output', type=str)
b.set_defaults(which='b')

args = main1.parse_args()
print(args)

# parse a second time with another parser
main2 = ArgumentParser(parents=[parent])
if args.which == 'a':
    main2.add_argument('a')
elif args.which == 'b':
    main2.add_argument('b')
    main2.add_argument('--output', type=str)

print(main2.parse_args())

Which gives:

$ ./tmp.py -vv a -v

Namespace(verbose=1, which='a')
Namespace(a='a', verbose=3)

$ ./tmp.py -vv b -v --output toto
Namespace(output='toto', verbose=1, which='b')
Namespace(b='b', output='toto', verbose=3)

$ ./tmp.py  -vv a  --output
usage: tmp.py [-h] [--verbose] {a,b} ...
tmp.py: error: unrecognized arguments: --output

I use this technique with multiple nested subparsers.

BlakBat
  • 1,835
  • 5
  • 17
  • 21