Argparse: Required arguments listed under "optional arguments"?

Question

I use the following simple code to parse some arguments; note that one of them is required. Unfortunately, when the user runs the script without providing the argument, the displayed usage/help text does not indicate that there is a non-optional argument, which I find very confusing. How can I get python to indicate that an argument is not optional?

Here is the code:

import argparse
if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description='Foo')
    parser.add_argument('-i','--input', help='Input file name', required=True)
    parser.add_argument('-o','--output', help='Output file name', default="stdout")
    args = parser.parse_args()
    print ("Input file: %s" % args.input )
    print ("Output file: %s" % args.output )

When running above code without providing the required argument, I get the following output:

usage: foo.py [-h] -i INPUT [-o OUTPUT]

Foo

optional arguments:
    -h, --help            show this help message and exit
    -i INPUT, --input INPUT
                          Input file name
    -o OUTPUT, --output OUTPUT
                          Output file name

In the usage line, the `-i INPUT` part is not surrounded by square brackets, which subtlety indicates that is indeed, required. Also, you can manually explain that through the `help` param — Jaime Rodríguez-Guerra, Jun 12 '14 at 09:15
@JaimeRGP Yes, but that's not sufficient, of course, and it's also less than prominent. The assigned group name `optional arguments` for the required arguments is still misleading. — Asclepius, Oct 05 '16 at 06:13

poke · Accepted Answer · 2016-10-05T06:53:48.993

Parameters starting with - or -- are usually considered optional. All other parameters are positional parameters and as such required by design (like positional function arguments). It is possible to require optional arguments, but this is a bit against their design. Since they are still part of the non-positional arguments, they will still be listed under the confusing header “optional arguments” even if they are required. The missing square brackets in the usage part however show that they are indeed required.

See also the documentation:

In general, the argparse module assumes that flags like -f and --bar indicate optional arguments, which can always be omitted at the command line.

Note: Required options are generally considered bad form because users expect options to be optional, and thus they should be avoided when possible.

That being said, the headers “positional arguments” and “optional arguments” in the help are generated by two argument groups in which the arguments are automatically separated into. Now, you could “hack into it” and change the name of the optional ones, but a far more elegant solution would be to create another group for “required named arguments” (or whatever you want to call them):

parser = argparse.ArgumentParser(description='Foo')
parser.add_argument('-o', '--output', help='Output file name', default='stdout')
requiredNamed = parser.add_argument_group('required named arguments')
requiredNamed.add_argument('-i', '--input', help='Input file name', required=True)
parser.parse_args(['-h'])

usage: [-h] [-o OUTPUT] -i INPUT

Foo

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Output file name

required named arguments:
  -i INPUT, --input INPUT
                        Input file name

I have been having the same issue. I tried you solution. It does add the arguments to the new group but my code doesn't seem to work after that. Any solutions would be appreciated. Link to my code - https://pastebin.com/PvC2aujz — Zarar Mahmud, Apr 13 '19 at 19:45
@ZararMahmud: You are passing in empty arguments in line 24 of your code: `parser.parse_args([])` Instead, use `parser.parse_args()` with no arguments to capture the contents of sys.argv. Per [argparse](https://docs.python.org/2/library/argparse.html#parsing-arguments) — Devin, Jun 12 '19 at 13:50
@poke: Nice solution! But this doesn't help in case you need mutual exclusive groups, or am I missing anything? — Judge, Jan 16 '20 at 15:40
@Judge i would recommend reading this https://pymotw.com/3/argparse/#mutually-exclusive-options — Peter Moore, Apr 23 '20 at 13:16
@mrgloom Options are usually defined with `--` and a precise name to explain what they are for. A single `-` can then be used to define a short alias for commonly used options. In my example, you could use both `--output out.txt` or `-o out.txt` for the exact same thing. The `-o` is just a short alias for `--output`. Some command line tools additionally allow you to chain these short aliases. E.g. `tar -xf archive.tar` is short for `tar --extract --file=archive.tar`. — poke, Jul 09 '20 at 22:39

score 131 · Answer 2 · edited Dec 14 '21 at 20:48

131

Since I prefer to list required arguments before optional, I hack around it via:

parser = argparse.ArgumentParser()
parser._action_groups.pop()
required = parser.add_argument_group('required arguments')
optional = parser.add_argument_group('optional arguments')
required.add_argument('--required_arg', required=True)
optional.add_argument('--optional_arg')
return parser.parse_args()

and this outputs:

usage: main.py [-h] --required_arg REQUIRED_ARG [--optional_arg OPTIONAL_ARG]

required arguments:
  --required_arg REQUIRED_ARG

optional arguments:
  --optional_arg OPTIONAL_ARG

I can live without -h, --help showing up in the optional arguments group.

edited Dec 14 '21 at 20:48

Antosha

344
4
10

answered Jan 19 '17 at 16:35

Karl Rosaen

4,508
2
27
30

3

Does this actually force argparse to treat any of the arguments as required? – Anthony Oct 12 '17 at 19:41
7

I think the 'required' argument still needs to be set when adding an argument. – Karl Rosaen Oct 13 '17 at 15:13
That's really nice. – Paul Cezanne Jan 12 '18 at 21:26
7

@Anthony - no you need the 'required=True' in add_argument for that. The above answer just illustrates argument grouping. – user2275693 Feb 28 '18 at 16:10
Now in the first line it looks as if the argument would be optional, so opposite of what OP shows. Any chance that required arguments are shown before optional ones AND have no brackets? – Isi Sep 30 '20 at 10:20
2

As noted in other answers, the use of private member `_action_groups` violates the API contract and risks breaking on the next minor release. – nclark Apr 07 '22 at 21:56

score 66 · Answer 3 · edited May 31 '19 at 15:25

66

Building off of @Karl Rosaen

parser = argparse.ArgumentParser()
optional = parser._action_groups.pop() # Edited this line
required = parser.add_argument_group('required arguments')
# remove this line: optional = parser...
required.add_argument('--required_arg', required=True)
optional.add_argument('--optional_arg')
parser._action_groups.append(optional) # added this line
return parser.parse_args()

and this outputs:

usage: main.py [-h] [--required_arg REQUIRED_ARG]
           [--optional_arg OPTIONAL_ARG]

required arguments:
  --required_arg REQUIRED_ARG

optional arguments:
  -h, --help                    show this help message and exit
  --optional_arg OPTIONAL_ARG

edited May 31 '19 at 15:25

fantabolous

21,470
7
54
51

answered Apr 17 '17 at 17:45

RalphyZ

843
8
9

1

BTW, are there any ways (methods) how to get access to `_action_group` without accessing protected member? In my case I need to add some argument to already existent (custom) group. – machin Apr 22 '17 at 09:21
This is great. Solves the --help item showing up in a second optional list. – Jeremy Nov 18 '17 at 19:17
3

**Note**: this answer breaks the exposed API, check [answer by Bryan_D](https://stackoverflow.com/a/57582191/6556360) down below. – lol May 02 '20 at 13:26

score 48 · Answer 4 · answered Aug 20 '19 at 22:19

One more time, building off of @RalphyZ

This one doesn't break the exposed API.

from argparse import ArgumentParser, SUPPRESS
# Disable default help
parser = ArgumentParser(add_help=False)
required = parser.add_argument_group('required arguments')
optional = parser.add_argument_group('optional arguments')

# Add back help 
optional.add_argument(
    '-h',
    '--help',
    action='help',
    default=SUPPRESS,
    help='show this help message and exit'
)
required.add_argument('--required_arg', required=True)
optional.add_argument('--optional_arg')

Which will show the same as above and should survive future versions:

usage: main.py [-h] [--required_arg REQUIRED_ARG]
           [--optional_arg OPTIONAL_ARG]

required arguments:
  --required_arg REQUIRED_ARG

optional arguments:
  -h, --help                    show this help message and exit
  --optional_arg OPTIONAL_ARG

Can you explain how RalphyZ's answer breaks the exposed API? — jeremysprofile, Sep 19 '19 at 15:46
`_action_groups` is intended for internal use only. Therefore, there is no compatibility guarantee across versions. — Bryan_D, Sep 20 '19 at 19:47
But then, [-h] is listed after positional arguments in usage message — Kiruahxh, May 02 '23 at 13:46

user2117647 · Answer 5 · 2021-05-17T00:35:07.993

You don't need to override the optional group.

Just do:

parser = argparse.ArgumentParser()
required = parser.add_argument_group('required arguments')
required.add_argument('--required_arg', required=True)
# All arguments set via parser directly will automatically go to the optional group
parser.add_argument('--optional_arg')
parser.print_help()

will print out

usage: [-h] --required_arg REQUIRED_ARG [--optional_arg OPTIONAL_ARG]

optional arguments:
  -h, --help            show this help message and exit
  --optional_arg OPTIONAL_ARG

required arguments:
  --required_arg REQUIRED_ARG

If you wish to have required arguments before optional, you can do the following:

parser = argparse.ArgumentParser()
optional = parser._action_groups.pop()
required = parser.add_argument_group('required arguments')
parser._action_groups.append(optional)
required.add_argument('--required_arg', required=True)
optional.add_argument('--optional_arg')
parser.print_help()

that will print groups in the correct order:

usage: [-h] --required_arg REQUIRED_ARG [--optional_arg OPTIONAL_ARG]

required arguments:
  --required_arg REQUIRED_ARG

optional arguments:
  -h, --help            show this help message and exit
  --optional_arg OPTIONAL_ARG

You're using a protected member `_action_groups` which puts you at serious risk of breaking on a minor release upgrade. I'll stick with @Karl Rosaen's answer. — nclark, Apr 07 '22 at 21:49
make that @poke's answer - all (useful) answers except his use `_action_groups`. — nclark, Apr 07 '22 at 21:58

atti · Answer 6 · 2022-07-14T23:09:56.070

NB: per Christophe Vu-Brugier, the code below does not work starting from Python version 3.10.

by default there're 2 argument groups in parser._action_groups: positional arguments and named arguments (titled 'optional arguments'). you can add your named optional arguments to the existing 'optional arguments' group, and required named arguments to a new 'required arguments' group. After that you can re-order groups:

import argparse

parser = argparse.ArgumentParser(description='Foo')

required = parser.add_argument_group('required arguments')

required.add_argument('-i','--input', help='Input file name', required=True)
parser.add_argument('-o','--output', help='Output file name', default="stdout")

groups_order = {
    'positional arguments': 0,
    'required arguments': 1,
    'optional arguments': 2
}
parser._action_groups.sort(key=lambda g: groups_order[g.title])

parser.parse_args(['-h'])

output:

usage: argparse_argument_groups.py [-h] -i INPUT [-o OUTPUT]

Foo

required arguments:
  -i INPUT, --input INPUT
                        Input file name

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Output file name

I used your snippet for a long time and it served me well. Thank you for that. However, I observe it no longer works with Python 3.10 because "optional arguments" was renamed to "options". Running this code triggers a "KeyError: options" exception on Python 3.10. Consequently, I decided it was safer for my project to drop the distinction between "required" and "optional" arguments. — Christophe Vu-Brugier, Jun 28 '22 at 09:33

score 0 · Answer 7 · answered Feb 07 '22 at 17:48

The required arguments are usually "positional" arguments when using argparse.

    parser = argparse.ArgumentParser(description="Foo")
    parser.add_argument("username")
    parser.add_argument("password")

This will add two positional arguments which are required.

Argparse: Required arguments listed under "optional arguments"?

7 Answers7

Linked