Is there a way to use doctest and sphinx to test and document command line applications?

Question

I have a Python module for which I'm writing a tutorial using Sphinx including doctests.

This module comes with a few helper programs.

I would like to include those helper programs in the documentation and have doctest check that the standard output is in sync between the current program version and the documentation.

I suppose I can use the sh module, or popen to check the standard output of a given program but I prefer that those tricks do not show up into the docs, or else non-programmers users will be certainly lost.

Is there a way to achieve that?

Possible duplicate of http://stackoverflow.com/questions/10886841/sphinx-and-argparse-autodocumenting-command-line-scripts — Steve Piercy, Apr 02 '17 at 09:39
I don't think this is a duplicate. The other question is solely about auto-documentation. This is primary about a much more interesting topic, using doctest to test command-line tools. — Raymond Hettinger, Jul 09 '17 at 23:57

Raymond Hettinger · Answer 1 · 2017-07-09T23:52:30.663

The doctest module only checks statements than can be run from the python interactive prompt.

Command-line tools can be invoked from the python interactive prompt using the subprocess module:

# Create Helper Function
>>> import subprocess
>>> run_commandline = lambda cmd: subprocess.check_output(cmd, shell=True).decode()

# Doctestable command-line calls
>>> print(run_commandline('cal 7 2017'))
     July 2017
Su Mo Tu We Th Fr Sa
                   1
 2  3  4  5  6  7  8
 9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31

>>> print(run_commandline('echo $BASH_VERSION'))
3.2.57(1)-release

There are likely some ways to hack doctest or sphinx to get what you want more directly, but this technique uses the advertised APIs for doctest, sphinx, and subprocess in exactly the ways they were designed to be used (doctest is designed to replay interactive prompt sessions found in docstrings, and subprocess is designed to run command-line tools directly from python and capture their output).

I suppose I can use the sh module, or popen to check the standard output of a given program but I prefer that those tricks do not show up into the docs, or else non-programmers users will be certainly lost.

Two thoughts: First, the details of those calls can mostly be hidden in a helper function to minimize distraction. Second, if you need to invoke command-line programs from Python, it isn't a trick to use popen or subprocess since those are the tools designed specifically for making those calls from Python.

score 0 · Answer 2 · answered Mar 21 '23 at 17:14

As Raymond Hettinger mentioned, you should create a function (e.g. shell) that accepts a string and run the corresponding string using the subprocess library. You can also decorate and manage the output stream (using contextlib.redirect_stdout) in order to make the result testable.

But, in the generated HTML, the same Python code is displayed and not the Shell one. In order to fix it, we used the following extension javascript code (which is based on copybutton.js):

$(document).ready(function() {
    const NAME_CLASS = "n";
    document.querySelectorAll(`.highlight-pycon pre .${NAME_CLASS}`).forEach(function(nameElement) {
        if (nameElement.innerText !== "shell")
            return;

        const GENERIC_PROMPT_CLASS = "gp";
        const promptElement = nameElement.previousElementSibling;
        if (!promptElement.classList.contains(GENERIC_PROMPT_CLASS))
            return;

        const GENERIC_OUTPUT_CLASS = "go";
        const GENERIC_TRACEBACK_CLASS = "gt";
        let pythonCode = "";
        let pythonCodeNodes = [];
        let currentNode = nameElement;
        while (
            currentNode
            && !(currentNode.classList?.contains(GENERIC_OUTPUT_CLASS)
                || currentNode.classList?.contains(GENERIC_TRACEBACK_CLASS))
        ) {
            pythonCode += currentNode.textContent;
            pythonCodeNodes.push(currentNode);
            currentNode = currentNode.nextSibling;
        }

        const outputStartElement = currentNode;

        const match = pythonCode.match(/shell\("(?<command>[^"]*)".*\)/);
        if (!match)
            return;

        const command = match.groups["command"]
        const invisiblePartsRemovedCommand = command.replace(/\s?\[.*\]/, '')
        const shellCode = invisiblePartsRemovedCommand + "\n"

        promptElement.innerText = "$ ";
        pythonCodeNodes.forEach(node => node.remove());
        promptElement.parentNode.insertBefore(document.createTextNode(shellCode), outputStartElement);
    });
});

Is there a way to use doctest and sphinx to test and document command line applications?

2 Answers2