30

I am using Sphinx to generate the documentation for a project of mine.

In this project, I describe a list of available commands in a yaml file which, once loaded, results in a dictionary in the form {command-name : command-description} for example:

commands = {"copy"  : "Copy the highlighted text in the clipboard",
            "paste" : "Paste the clipboard text to cursor location",
            ...}

What I would like to know, is if there is a method in sphinx to load the yaml file during the make html cycle, translate the python dictionary in some reStructuredText format (e.g. a definition list) and include in my html output.

I would expect my .rst file to look like:

Available commands
==================
The commands available in bla-bla-bla...

.. magic-directive-that-execute-python-code::
   :maybe python code or name of python file here:

and to be converted internally to:

Available commands
==================
The commands available in bla-bla-bla...

copy
  Copy the highlighted text in the clipboard

paste
  Paste the clipboard text to cursor location

before being translated to HTML.

bad_coder
  • 11,289
  • 20
  • 44
  • 72
mac
  • 42,153
  • 26
  • 121
  • 131
  • This is not a proper answer so I put it as a comment. As far as I know, there's no way to parse a yaml file with sphinx directly, but I think you can make it using pyyaml and modifying you sphinx Makefile. – CastleDweller Aug 30 '11 at 23:00
  • 1
    What's the point of writing YAML code? Why not just write the description in the Python module and use Sphinx's autodoc? Why make something that's *more* complicated than http://sphinx.pocoo.org/ext/autodoc.html? – S.Lott Aug 30 '11 at 23:29
  • @S.Lott - The basic idea is DNRY: commands are defined (and can be overridden by the user) in a yml file. The example above is simplified for making the question easier to understand, but the actual yml file contains indeed extra information for the parser like the number of parameters, the possible flags, the validation callback, etc... It simply seems stupid (and a potential source of bugs in the documentation) to repeat the same information in the yml file and the module docstring. – mac Aug 31 '11 at 06:33
  • 1
    @Oscar - Appreciate you took the time to drop a comment, but pyyaml is not the problem here (obviously I'm using it already in the module that loads the yml file). The problem is that I don't know how I should modify the makefile... would you mind to articulate your suggestion a bit more? Thanks! – mac Aug 31 '11 at 06:37
  • 1
    @mac: Why waste time writing code in YAML? If you want the users to be able to define (and override) commands, Python is perfectly acceptable. The Python syntax for a dictionary is no more complex than YAML, and you simply avoid this weird multi-language issue. – S.Lott Aug 31 '11 at 12:04
  • @S.Lott - The problem has nothing to do with the original information being serialised with YAML. The problem is about including rst generated with python "on the fly" (at doc-build time). I solved it now however, will post an answer to my own question later on... – mac Aug 31 '11 at 14:45
  • @mac: "including rst generated with python "on the fly" (at doc-build time)" doesn't seem to be part of the question at all. Could you please **update** the question to make that clear. The question seems to be about using YAML instead of Python code and documenting the YAML as if it was Python code. – S.Lott Aug 31 '11 at 14:54

7 Answers7

27

At the end I find a way to achieve what I wanted. Here's the how-to:

  1. Create a python script (let's call it generate-includes.py) that will generate the reStructuredText and save it in the myrst.inc file. (In my example, this would be the script loading and parsing the YAML, but this is irrelevant). Make sure this file is executable!!!

  2. Use the include directive in your main .rst document of your documentation, in the point where you want your dynamically-generated documentation to be inserted:

    .. include:: myrst.inc

  3. Modify the sphinx Makefile in order to generate the required .inc files at build time:

    myrst.inc:
    ./generate-includes.py

html: myrst.inc
    ...(other stuff here)

  4. Build your documentation normally with make html.

mac
  • 42,153
  • 26
  • 121
  • 131
  • 1
    Works great. Althought readthedocs doesn't work anymore, but I guess that's inevitable for python-generated documentation... – Mark Jun 05 '15 at 21:20
18

An improvement based on Michael's code and the built-in include directive:

import sys
from os.path import basename

try:
    from StringIO import StringIO
except ImportError:
    from io import StringIO

from docutils.parsers.rst import Directive    
from docutils import nodes, statemachine

class ExecDirective(Directive):
    """Execute the specified python code and insert the output into the document"""
    has_content = True

    def run(self):
        oldStdout, sys.stdout = sys.stdout, StringIO()

        tab_width = self.options.get('tab-width', self.state.document.settings.tab_width)
        source = self.state_machine.input_lines.source(self.lineno - self.state_machine.input_offset - 1)

        try:
            exec('\n'.join(self.content))
            text = sys.stdout.getvalue()
            lines = statemachine.string2lines(text, tab_width, convert_whitespace=True)
            self.state_machine.insert_input(lines, source)
            return []
        except Exception:
            return [nodes.error(None, nodes.paragraph(text = "Unable to execute python code at %s:%d:" % (basename(source), self.lineno)), nodes.paragraph(text = str(sys.exc_info()[1])))]
        finally:
            sys.stdout = oldStdout

def setup(app):
    app.add_directive('exec', ExecDirective)

This one imports the output earlier so that it goes straight through the parser. It also works in Python 3.

chendanlan
  • 23
  • 5
alex.forencich
  • 1,355
  • 11
  • 16
  • 2
    With Sphinx v1.8.1, this code generates `Extension error: ... cannot import name 'Directive' from 'sphinx.util.compact'`. Changing the import to `from docutils.parsers.rst import Directive` fixes the issue. – AJNeufeld Oct 16 '18 at 15:43
9

I needed the same thing, so I threw together a new directive that seems to work (I know nothing about custom Sphinx directives, but it's worked so far):

import sys
from os.path import basename
from StringIO import StringIO

from sphinx.util.compat import Directive
from docutils import nodes

class ExecDirective(Directive):
    """Execute the specified python code and insert the output into the document"""
    has_content = True

    def run(self):
        oldStdout, sys.stdout = sys.stdout, StringIO()
        try:
            exec '\n'.join(self.content)
            return [nodes.paragraph(text = sys.stdout.getvalue())]
        except Exception, e:
            return [nodes.error(None, nodes.paragraph(text = "Unable to execute python code at %s:%d:" % (basename(self.src), self.srcline)), nodes.paragraph(text = str(e)))]
        finally:
            sys.stdout = oldStdout

def setup(app):
    app.add_directive('exec', ExecDirective)

It's used as follows:

.. exec::
   print "Python code!"
   print "This text will show up in the document"
Michael Mrozek
  • 169,610
  • 28
  • 168
  • 175
  • 1
    With Sphinx v1.8.1, this code generates `Extension error: ... cannot import name 'Directive' from 'sphinx.util.compact'`. Changing the import to `from docutils.parsers.rst import Directive` fixes the issue. – AJNeufeld Oct 16 '18 at 15:43
4

Sphinx doesn't have anything built-in to do what you like. You can either create a custom directive to process your files or generate the reStructuredText in a separate step and include the resulting reStructuredText file using the include directive.

devin_s
  • 3,345
  • 1
  • 27
  • 32
  • So good to know that `.. magic-directive-that-execute-python-code::` needs to be something I code then. I tried to get a crack at it and although I manage to write a directive that insert my script-generated text I couldn't understand how I can make it parse. For example: `nodes.paragraph('', '**text**')` will output `**text**` rather than **`text`**. How can I tell sphinx to parse it with standard *reStructuredText* syntax? – mac Aug 31 '11 at 09:03
  • In your Directive subclass you'll have handle_content() and handle_signature() methods. You can recursively call self.state.nested_parse() and it will handle the built-in styles properly. Checkout [Creating reStructuredText Directives](http://docutils.sourceforge.net/docs/howto/rst-directives.html) – devin_s Aug 31 '11 at 16:37
  • Thank you devin_s (+1). As you may have noticed (see my own answer) I finally solved using the `.. include::` directive. Will check out your solution too, though, asap! – mac Aug 31 '11 at 18:15
3

I know this question is old, but maybe someone else will find it useful as well.

It sounds like you don't actually need to execute any python code, but you just need to reformat the contents of your file. In that case you might want to look at sphinx-jinja (https://pypi.python.org/pypi/sphinx-jinja).

You can load your YAML file in the conf.py:

jinja_contexts = yaml.load(yourFileHere)

Then you can use jinja templating to write out the contents and have them treated as reST input.

Peter
  • 61
  • 2
1

Sphinx does support custom extensions that would probably be the best way to do this http://sphinx.pocoo.org/ext/tutorial.html.

Matti Pastell
  • 9,135
  • 3
  • 37
  • 44
0

Not quite the answer you're after, but perhaps a close approximation: yaml2rst. It's a converter from YAML to RST. Doesn't do anything explicitly fancy with the YAML itself, but looks for comment lines (starts with #) and pulls them out into RST chunks (with the YAML going into code-blocks). Allows for a sort-of literate YAML.

Also, the syntax-highlighted YAML is quite readable (heck, it's YAML, not JSON!).

dsz
  • 4,542
  • 39
  • 35