Python argparse REMAINDER is not clear

Question

As documentation suggests:

argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:

>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print parser.parse_args('--foo B cmd --arg1 XX ZZ'.split())
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')

I tried to use this to exactly the same purpose, but in some circumstances it seems buggy for me (or perhaps I get the concept wrong):

import argparse

a = argparse.ArgumentParser()

a.add_argument('-qa', nargs='?')
a.add_argument('-qb', nargs='?')
a.add_argument('rest', nargs=argparse.REMAINDER)

a.parse_args('-qa test ./otherutil bar -q atr'.split())

Result:

test.py: error: ambiguous option: -q could match -qa, -qb

So apparently, if the otherutil has such arguments which somehow "collide" with the arguments given to argparse, it doesn't seem to work correctly.

I would expect when argparse reaches the REMAINDER kind of argument, it just uses up all the strings in the end of the list without any further parsing. Can I reach this effect somehow?

Old question, but adding test as a sub parser should work on this. the -q for that would then be handled by that sub-parser rather than the top-level one. I'd also use --qa and --qb — Kevin Shea, Jun 29 '17 at 11:41
Don't use single-dash multi-letter options, like `-qa` (but `--qa`), and your're done, Q had nothing to do with `REMINDER`. @hpaul [nailed it](https://stackoverflow.com/a/56640098/548792). — ankostis, May 08 '23 at 14:16

score 9 · Answer 1 · answered Aug 04 '18 at 15:01

I ran into this while trying to dispatch options to an underlying utility. The solution I wound up using was nargs='*' instead of nargs=argparse.REMAINDER, and then just use the "pseudo-argument" -- to separate the options for my command and the underlying tool:

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--myflag', action='store_true')
>>> parser.add_argument('toolopts', nargs='*')
>>> parser.parse_args('--myflag -- -a --help'.split())
Namespace(myflag=True, toolopts=['-a', '--help'])

This is reasonably easy to document in the help output.

score 6 · Answer 2 · answered Jun 17 '19 at 23:44

This has more to do with handling of abbreviations than with the REMAINDER nargs.

In [111]: import argparse                                                                 
In [112]: a = argparse.ArgumentParser() 
     ...:  
     ...: a.add_argument('-qa', nargs='?') 
     ...: a.add_argument('-qb', nargs='?')                                                

In [113]: a.parse_args('-qa test ./otherutil bar -q atr'.split())                         
usage: ipython3 [-h] [-qa [QA]] [-qb [QB]]
ipython3: error: ambiguous option: -q could match -qa, -qb

argparse does a 2 pass parsing. First it tries to categorize the strings as options (flags) or arguments. Second it alternates between parsing positionals and optionals, allocating arguments according to the nargs.

Here the ambiguity occurs in the first pass. It's trying to match '-q' with the two available optionals. REMAINDER's special action (absorbing '-q' as though it were an plain string) doesn't occur until the second pass.

Newer argparse versions allow us to turn off the abbreviation handling:

In [114]: a.allow_abbrev                                                                  
Out[114]: True
In [115]: a.allow_abbrev=False                                                            
In [116]: a.parse_args('-qa test ./otherutil bar -q atr'.split())                         
usage: ipython3 [-h] [-qa [QA]] [-qb [QB]]
ipython3: error: unrecognized arguments: ./otherutil bar -q atr

And if I add the REMAINDER action:

In [117]: a.add_argument('rest', nargs=argparse.REMAINDER) 

In [118]: a.parse_args('-qa test ./otherutil bar -q atr'.split())                         
Out[118]: Namespace(qa='test', qb=None, rest=['./otherutil', 'bar', '-q', 'atr'])

The use of '--' as @Colin suggests works because that string is recognized in the first pass:

In [119]: a.allow_abbrev=True                                                             
In [120]: Out[117].nargs='*'                                                              
In [121]: a.parse_args('-qa test -- ./otherutil bar -q atr'.split())                      
Out[121]: Namespace(qa='test', qb=None, rest=['./otherutil', 'bar', '-q', 'atr'])

score 4 · Answer 3 · edited May 23 '17 at 12:33

You need to use two --.

a.add_argument('--qa', nargs='?')
a.add_argument('--qb', nargs='?')

So the options that you define collide with a -q, that accepts at least an argument, defined somewhere else

From argparse doc

ArgumentParser.add_argument(name or flags...)

name or flags - Either a name or a list of option strings, e.g. foo or -f, --foo.

EDIT to reply to @PDani first comment:

This post is interesting.

From what I have understood, argparse follows the POSIX and GNU style.

An important thing is that short (1 letter) option can be grouped together and if one option required one argument this can be be attached to the option letter. For example if you have something like this

a.add_argument('-a', action='store_true')
a.add_argument('-b', action='store_true')
a.add_argument('-c', action='store_true')
a.add_argument('-d', nargs=1)
a.add_argument('-e', nargs=1)

you can call them as -abcd3 -e5 or -a -b -c -d3 -e5 or -cba -e5 -d3, ...
Now, if you have

a.add_argument('-abc',  action='store_true')

and you have would be very hard for argparse to decide if -abc is 3 short arguments attached or one long. So you are forced to define the argument as --abc.

So I guess that you can't use long arguments name with one -.

I know of an alternative way to do the command line parsing called docopt: you can give a look but I doubt that it can solve your problem.

so if I understand correctly, argparse unable to handle two-letter "short arguments"? Actually my aim is to mimic the behavior of an existing command line tool with argparse, which uses options like "-qsomething" and "-qotherthing" (where something and otherthing aren't optional arguments). — PDani, Mar 18 '13 at 15:43
Responding to @PDani's old comment. The supported way to do this in 2021 is: `parser.add_argument("-q",dest="qq",choices=["something","otherthing"])` — DanHorner, Apr 12 '21 at 16:40

score 4 · Answer 4 · answered Dec 14 '15 at 21:48

Perhaps some combination of ArgumentParser.parse_known_args() and some other bits of special handling?

This is not be perfect, but might lead in the right direction:

import argparse
import sys

a = argparse.ArgumentParser()
# treat the common-prefixed arguments as options to the prefix
a.add_argument("-q")

# allow a delimiter to set off your arguments from those which should go to the
# other utility, and use parse_known_args() if the delimiter is not present
argv = sys.argv[1:]
if "--" in argv:
    i = argv.index("--")
    args, extra = a.parse_args(argv[:i]), argv[i + 1:]
else:
    a.add_argument("extra", nargs=argparse.REMAINDER)
    args, _ = a.parse_known_args(argv)
    extra = args.extra

# complain if the `-q` option was not specified correctly
if args.q not in ("something", "otherthing"):
    a.error("Must specify '-qsomething' or '-qotherthing'")

print "q:", "-q%s" % (args.q,)
print "extra:", '"%s"' % (" ".join(extra),)

Result:

$ ./testcmd -qsomething test ./otherutil bar -q atr
q: -qsomething
extra: "test ./otherutil bar -q atr"

Caveats:

This will allow a space between -q and the rest of the -q-prefixed option.
This will consume one -q option, but I don't recall whether it will raise an exception (or do any other helpful thing) if more are specified.

This helped me a lot. The splitting on `'--'` before passing to argparse makes so much sense. I was about to go dealing with custom actions etc. Thank you. — Andy Lester, Dec 12 '19 at 04:07

Python argparse REMAINDER is not clear

4 Answers4