Using variable arg names with argparse

Question

I have an upstream system that invokes my program with varying arg names. Example:

foo --xyz1 10 --xyz2 25 --xyz3 31

I would like the result of argparsing to be a xyz = [10, 25, 31].

The names of my args have a common prefix, but unfortunately have to differ at least with a different numeric suffix, which also indicates order. I also don't have a fixed number of args.

Is there a way to model this with argparse? Either with what is available through some combination of built-in capabilities, or by overriding/pluging in some custom parser handling.

`action='append'` lets you reuse the `--xyz` flag, and collect values in the order that they were given, e.g. '--xyz 10 --xyz 25 --xyz 31' — hpaulj, Feb 19 '17 at 17:32
I need to use varying arg names; thats a requirement from another part of the system. So I need to handle --xyz1, --xyz2, --xyz3, and so on. — Nikhil Kothari, Feb 19 '17 at 17:49
Unknown ahead of time. In practical scenarios it will be less than 100... but it could be 1, 2, ... N. Enough that I don't really want to add N literal args, and rely on some of them being specified. — Nikhil Kothari, Feb 19 '17 at 17:58
`argparse` is not the right parser for this case. Write your own using `sys.argv`. — hpaulj, Feb 19 '17 at 18:01
How about partial parsing (https://docs.python.org/3/library/argparse.html#partial-parsing) and then manually going through the list of unknown args to check them for the required prefix? — Oliver Dain, Feb 19 '17 at 18:46
Could you change the format to be `--xyz 1 10 --xyz 2 25 --xyz 3 31`? Use `nargs=2` then: `args.xyz = [value for _, value in args.xyz]` — Bakuriu, Feb 19 '17 at 20:03

Stephen Rauch · Accepted Answer · 2017-02-19T20:12:51.943

I would suggest a bit of pre-processing to achieve this:

Code:

def get_xyz_cmd_line(xyz_cmd_line):
    # build a generator to iterate the cmd_line
    cmd_line_gen = iter(xyz_cmd_line)

    # we will separate the xyz's from everything else
    xyz = []
    remaining_cmd_line = []

    # go through the command line and extract the xyz's
    for opt in cmd_line_gen:
        if opt.startswith('--xyz'):
            # grab the opt and the arg for it
            xyz.append((opt, cmd_line_gen.next()))
        else:
            remaining_cmd_line.append(opt)

    # sort the xyz's and return all of them as -xyz # -xyz # ... 
    return list(it.chain(*[
        ('--xyz', x[1]) for x in sorted(xyz)])) + remaining_cmd_line

To Test:

import argparse
import itertools as it

parser = argparse.ArgumentParser(description='Get my Option')
parser.add_argument('--an_opt', metavar='N', type=int,
                    help='An option')
parser.add_argument('--xyz', metavar='N', type=int, action='append',
                    help='An option')

cmd_line = "--an_opt 1 --xyz1 10 --xyz3 31 --xyz2 25 ".split()
args = parser.parse_args(get_xyz_cmd_line(cmd_line))
print(args)

Output:

Namespace(an_opt=1, xyz=[10, 25, 31])

To use:

Nominally instead of a fixed cmd_line as in the above example this would be called with something like:

args = parser.parse_args(get_xyz_cmd_line(sys.argv[1:]))

UPDATE: If you need --xyz=31 (ie = separator):

Then you will need to change:

# grab the opt and the arg for it
xyz.append((opt, cmd_line_gen.next()))

To:

if '=' in opt:
    xyz.append(tuple(opt.split('=', 1)))
else:
    # grab the opt and the arg for it
    xyz.append((opt, cmd_line_gen.next()))

I like your solution relative to mine because mine (posted below relied on an internal implementation detail). However, I don't yet know if I can use it, as I have one component adding the argument definitions, and the user calling parse_args, and I'd like to have the user not know to do, or even which args to perform preprocessing for. With my derived argparser, and custom action, this scenario is accomplished without any user intervention. — Nikhil Kothari, Feb 19 '17 at 21:10

Nikhil Kothari · Answer 2 · 2017-02-19T21:19:29.163

Here's what I did for reference (quick and dirty version), though I also like Stephen Rauch's answer (so I'll mark that as an answer -- esp. since I used internal implementation detail for my solution):

class CustomArgumentsParser(argparse.ArgumentParser):

  def _parse_optional(self, arg_string):
    suffix_index = arg_string.find(':')
    if suffix_index < 0:
      return super(CustomArgumentParser, self)._parse_optional(arg_string)

    original_arg_string = arg_string
    suffix = arg_string[suffix_index + 1:]
    arg_string = arg_string[0:suffix_index]

    option_tuple = super(CustomArgumentParser, self)._parse_optional(arg_string)
    if not option_tuple:
      return option_tuple

    action, option_string, explicit_arg = option_tuple
    if isinstance(action, BuildListAction):
      return action, suffix, explicit_arg
    else:
      self.exit(-1, message='Unknown argument %s' % original_arg_string)


class BuildListAction(argparse.Action):
  def __init__(self,
               option_strings,
               dest,
               nargs=None,
               const=None,
               default=None,
               type=None,
               choices=None,
               required=False,
               help=None,
               metavar=None):
    super(BuildListAction, self).__init__(
      option_strings=option_strings,
      dest=dest,
      nargs=nargs,
      const=const,
      default=default,
      type=type,
      choices=choices,
      required=required,
      help=help,
      metavar=metavar)

  def __call__(self, parser, namespace, values, option_string=None):
    index = int(option_string) - 1

    list = getattr(namespace, self.dest)
    if list is None:
      list = []
      setattr(namespace, self.dest, list)

    if index >= len(list):
      list.extend([self.default] * (index + 1 - len(list)))
    list[index] = values

Usage:

argparser = CustomArgumentsParser()
argparser.add_argument('--xyz', type=int, action=BuildListAction)

Note -- This supports args of the form --xyz:1, --xyz:2, ... which is slightly different than the original question.

Using variable arg names with argparse

2 Answers2

Linked