Best architecture for a Python command-line tool with multiple subcommands

Question

I am developing a command-line toolset for a project. The final tool shall support many subcommands, like so

foo command1 [--option1 [value]?]*

So there can be subcommands like

foo create --option1 value --

foo make file1 --option2 --option3

The tool uses the argparse library for handling command-line arguments and help functionality etc.

A few additional requirements and constraints:

• Some options and functionality is identical for all subcommands (e.g. parsing a YAML configuration file etc.)

• Some subcommands are quick and simple to code, because they e.g. just call an external bash script.

• Some subcommands will be complex and hence long code.

• Help for the basic tool should be available as well as for an individual subcommand:

  foo help Available commands are: make, create, add, xyz

  foo help make Details for the make subcommand

• error codes should be uniform across the subcommands (like the same error code for "file not found")

For debugging purposes and for making progress with self-contained functionality for minimal viable versions, I would like to develop some subcommands as self-containted scripts and modules, like

make.py

that can be imported into the main foo.py script and later on invoked as both

make.py --option1 value etc.

and

foo.py make --option1 value

Now, my problem is: What is the best way to modularize such a complex CLI tool with minimal redundancy (e.g. the arguments definition and parsing should only be coded in one component)?

Option 1: Put everything into one big script, but that will become difficult to manage.

Option 2: Develop the functionality for a subcommand in individual modules / files (like make.py, add.py); but such must remain invocable (via if __name__ == '__main__' ...).

The functions from the subcommand modules could then be imported into the main script, and the parser and arguments from the subcommand added as a subparser.

Option 3: The main script could simply reformat the call to a subcommand to subprocess, like so

subprocess.run('./make.py {arguments}', shell=True, check=True, text=True)

Answer 1

I'm more used to answering questions about the details of numpy and argparse, but here's how I envisage a large package.

In a main.py:

import submod1
# ....
sublist = [submod1, ...]
def make_parser(sublist):
    parser = argparse.ArgumentParser()
    # parser.add_argument('-f','--foo')  # main specific
    # I'd avoid positionals
    sp = parser.add_subparsers(dest='cmd', etc)
    splist=[]
    for md in sublist:
         sp1 = sp.add_parser(help='', parents=[md.parser])
         sp1.set_default(func=md.func)  # subparser func as shown in docs
         splist.append(sp1)
    return parser

if name == 'main': parser = make_parser(sublist) args = parser.parse_args() # print(args) # debugging display args.func(args) # again the subparser func

In submod1.py

import argparse def make_parser(): parser = argparse.ArgumentParser(add_help=False) # check docs? parser.add_argument(...) # could add a common parents here return parser

parser.make_parser()

def func(args):
    # module specific 'main'

I'm sure this is incomplete in many ways, since I've written this on the fly without testing. It's a basic subparser definition as documented, but using parents to import subparsers as defined in the submodules. parents could also be used to define common arguments for subparsers; but utility functions would work just as well. I think parents is most useful when using a parser that you can't otherwise access; ie. an imported one.

parents essentially copies Actions from one parser to the new one - copy by reference (not by value or as a copy). It is not a highly developed tool, and there have been a number of SO where people ran into problems. So don't try to over extend it.

Answer 2

Consider using the Command Pattern along with the Factory Method Pattern.

In short, create an abstract class called Command and make each command it's own class inheriting from Command.

Example:

class Command():

    def execute(self):
        raise NotImplementedError()


class Command1(Command):

    def __init__(self, *args):
        pass

    def execute(self):
        pass


class Command2(Command):

    def __init__(self, *args):
        pass

    def execute(self):
        pass

This will handle execution of commands. For building, make a command factory.

class CommandFactory():

    @staticmethod
    def create(command, *args):
        if command == 'command1':
            return Command1(args)
        elif command == 'command2':
            return Command2(args)

Then you'd be able to execute a command with one line:

CommandFactory.create(command, args).execute()

Answer 3

Thanks for all of your suggestions!

I think the most elegant approach is using Typer and following this recipe:

https://typer.tiangolo.com/tutorial/subcommands/add-typer/