Is sphinxcontrib-autoprogram parsing arguments after grabbing the parser?

Question

I am creating a Python package with a command-line interface that uses the subcommand pattern: kevlar count, kevlar partition, and so on. The CLI works wonderfully as it is, and now I'm trying to add the CLI to my Sphinx documentation. In searching for a solution, I came across sphinxcontrib-autoprogram which seems to do exactly what I want, even explicitly handling subcommands. But when I execute the sphinx build, I get the following error.

sphinx-build -b html -d _build/doctrees   . _build/html
Running Sphinx v1.6.3
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 5 source files that are out of date
updating environment: 5 added, 0 changed, 0 removed
reading sources... [ 20%] cli
usage: sphinx-build [-h] [-v] [-l F] cmd ...
sphinx-build: error: argument cmd: invalid choice: 'html' (choose from 'reaugment', 'dump', 'novel', 'collect', 'mutate', 'assemble', 'filter', 'partition', 'count', 'localize')
make[1]: *** [html] Error 2
make: *** [doc] Error 2

It seems like the sphinx extension is not only creating the argparse object (expected), but is also calling parse_args() on it (unexpected). The "invalid" html argument comes from the sphinx command-line build invocation, but is being mistaken somewhere as one of the subcommands from my library's CLI.

My syntax seems to match the sphinxcontrib-autoprogram documentation.

.. autoprogram:: cli:parser
   :prog: kevlar

What could be causing this behavior?

I'm not sure if these details are relevant to the issue, but in case they are:

the parser is defined in kevlar/cli/__init__.py:parser()
the subparsers for each command are defined in dedicated files in kevlar/cli/ (for example kevlar/cli/count.py)

I'm not sure if it helps, but this problem reminds me of https://stackoverflow.com/q/6912025/407651 (which is about optparse rather than argparse). — mzjn, Jul 18 '17 at 06:58

score 2 · Answer 1 · answered Jul 24 '17 at 21:48

You should load the argparse instance in a separate module that create the same argparse instance but doesn't execute the parser itself. Or your module could detect that it was loaded within autoprogram and exit after the argparse instance was constructed.

For example the PoC-Library uses a very huge argparse command line parser with lots of sub-parsers. The front-end script is this: py/PoC.py

The docs directory contains a dummy front-end, that triggers an instantiation of argparse, but aborts after its construction.

Code to dummy load PoC:

from sys import path as sys_path
sys_path.append("../py")

from PoC import PileOfCores

# entry point
parser = PileOfCores(False, False, False, True, sphinx=True).MainParser

Source: docs/PoCSphinx.py

Code to load and abort if loaded by Sphinx:

def __init__(self, debug, verbose, quiet, dryRun, sphinx=False):
    # Call the initializer of ILogable
    # --------------------------------------------------------------------------
    if quiet:      severity = Severity.Quiet
    elif debug:    severity = Severity.Debug
    elif verbose:  severity = Severity.Verbose
    else:          severity = Severity.Normal

    logger = Logger(severity, printToStdOut=True)
    ILogable.__init__(self, logger=logger)

    # Call the constructor of the ArgParseMixin
    # --------------------------------------------------------------------------
    description = dedent("""\
        This is the PoC-Library Service Tool.
        """)
    epilog = "Pile-of-Cores"

    class HelpFormatter(RawDescriptionHelpFormatter):
        def __init__(self, *args, **kwargs):
            kwargs['max_help_position'] = 25
            super().__init__(*args, **kwargs)

    ArgParseMixin.__init__(self, description=description, epilog=epilog, formatter_class=HelpFormatter, add_help=False)
    if sphinx: return

Source: py/PoC.py

The class PileOfCores implements a property to return the main-parser object MainParser, which is stored in the variable parser expected by autoprogram.

score 0 · Answer 2 · answered Jul 18 '17 at 00:06

First, ensure your program uses argparse, which is a requirement for autoprogram:

scans argparse.ArgumentParser object, and then expands it into a set of .. program:: and .. option:: directives.

Second, the syntax you used might not be correct. It looks like you copy-pastad from the first example instead of reading its usage. Specifically:

.. autoprogram:: module:parser

module is the dotted import name of the module, and parser is a variable that refers to an argparse.ArgumentParser object or a Python expression that creates and returns one.

Thus in your case, assuming that your parser() creates and returns an argparse.ArgumentParser, your syntax would be something like this or close to it:

.. autoprogram:: kevlar.cli:parser()
    :prog: kevlar

The hard part is figuring out the exact, correct module:parser substitution.

For comparing against another example, see the source program, source reST file, and rendered HTML of the Pyramid documentation of pcreate.

"The hard part is figuring out the exact, correct `module:parser` substitution." Umm, the error message makes it pretty clear that the correct function is being called. The problem is that the `ArgumentParser` object seems to be *executed* rather than *inspected*. — Daniel Standage, Jul 18 '17 at 19:03

Is sphinxcontrib-autoprogram parsing arguments after grabbing the parser?

2 Answers2