Sphinx cannot find my python files. Says 'no module named ...'

Question

I have a question regarding the Sphinx autodoc generation. I feel that what I am trying to do should be very simple, but for some reason, it won't work.

I have a Python project of which the directory is named slotting_tool. This directory is located at C:\Users\Sam\Desktop\picnic-data-shared-tools\standalone\slotting_tool

I set up Sphinx using sphinx-quickstart. Then my directory structure (simplified) is as follows:

slotting_tool/
|_ build/
|_ source/
|___ conf.py
|___ index.rst
|_ main/
|___ run_me.py

Now, I set the root directory of my project to slotting_tool by adding the following to the conf.py file.

import os
import sys
sys.path.insert(0, os.path.abspath('..'))

Next, I update my index.rst file to look like this:

.. toctree::
   :maxdepth: 2
   :caption: Contents:

.. automodule:: main.run_me
   :members:

When trying to build my html using the sphinx-build -b html source .\build command, I get the following output, with the no module named error:

(base) C:\Users\Sam\Desktop\picnic-data-shared-tools\standalone\slotting_tool>sphinx-build -b html source .\build
Running Sphinx v1.8.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [] 0 added, 1 changed, 0 removed
reading sources... [100%] index
WARNING: autodoc: failed to import module 'run_me' from module 'main'; the following exception was raised:
No module named 'standalone'
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 1 warning.

The HTML pages are in build.

There are no HTML pages that refer to run_me.py in build. I have tried setting my root directory to all different kinds of directories and I have tried replacing all dots . with backslashes \ and so forth, but can't seem to find out what I'm doing wrong.

By the way, the statement that standalone is not a module is in fact true, it is just a directory without an __init__.py. Don't know if that might have caused some trouble?

Anyone have an idea?

Answer 1

This is the usual "canonical approach" to "getting started" applied to the case when your source code resides in a src directory like Project/src instead of simply being inside the Project base directory.

Follows these steps:

1. Create a docs directory in your Project directory (it's from this docs directory the commands in the following steps are executed).

2. sphinx-quickstart (choose separate source from build. Places .html and .rst files in different folders).

3. sphinx-apidoc -o ./source ../src

4. make html

This would yield the following structure (provided you .py source files reside in Project/src):

Project
|
├───docs
│   │   make.bat
│   │   Makefile
│   │
│   ├───build
│   └───source
│       │   conf.py
│       │   index.rst
│       │   modules.rst
│       │   stack.rst
│       │
│       ├───_static
│       └───_templates
└───src
        stack.py

In your conf.py you'd add (after step 2):

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join('..', '..', 'src')))

Also include in conf.py:

extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon']

And in index.rst you'd link modules.rst:

Welcome to Project's documentation!
================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules
      
   
Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

Your stack.rst and modules.rst were auto-generated by sphinx-apidoc, no need to change them (at this point). But just so you know this is what they look like:

stack.rst:

stack module
============

.. automodule:: stack
   :members:
   :undoc-members:
   :show-inheritance:

modules.rst:

src
===

.. toctree::
   :maxdepth: 4

   stack

---

---

After `make html` open `Project/docs/build/index.html` in your browser, the results:

and:

Answer 2

Let's take an example with a project: dl4sci-school-2020 on master branch, commit: 6cbcc2c72d5dc74d2defa56bf63706fd628d9892:

├── dl4sci-school-2020
│   ├── LICENSE
│   ├── README.md
│   ├── src
│   │   └── __init__.py
│   └── utility
│       ├── __init__.py
│       └── utils.py

and utility package has a utils.py module:

Follow this process(FYI, I'm using sphinx-build 3.1.2):

1. create a docs/ directory under you project:

mkdir docs
cd docs

2. start sphinx within docs/, and just pass your project_name, your_name & version of your choice and rest keep defaults.

sphinx-quickstart

you will get below auto-generated in your docs/ folder

├── docs
│   ├── Makefile
│   ├── build
│   ├── make.bat
│   └── source
│       ├── _static
│       ├── _templates
│       ├── conf.py
│       └── index.rst

Since, we created a separate docs directory so we need sphinx find where to find build files and python src module. So, edit the conf.py file, you can use my conf.py file too

import os
import sys
basedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
sys.path.insert(0, basedir)

Now, to enable access to nested multiple packages & modules if any, you need to edit index.rst file.

.. toctree::
   :maxdepth: 2
   :caption: Description of my CodeBase:

   modules

The modules picks up content from modules.rst file which we will create below: Make sure you're still in doc/ to run the below command

sphinx-apidoc -o ./source ..

The output you get:

├── docs
│   ├── Makefile
│   ├── build
│   ├── make.bat
│   └── source
│       ├── _static
│       ├── _templates
│       ├── conf.py
│       ├── index.rst
│       ├── modules.rst
│       ├── src.rst
│       └── utility.rst

now run:

make html

Now, go and open in browser of your choice,

file:///<absolute_path_to_your_project>/dl4sci-school-2020/docs/build/html/index.html

have you beautiful documentation ready

https://i.stack.imgur.com/5pvLu.jpg

FYI, You can switch any theme of your choice, I found sphinx_rtd_theme and extension sphinxcontrib.napoleon super dope!. Thanks to their creators, so I used it.

below does the work!

pip install sphinxcontrib-napoleon
pip install sphinx-rtd-theme

You can host your documentation it on readthedocs enjoy documenting your code!

Answer 3

sys.path.insert(0, os.path.abspath('../..'))

That's not correct. Steve Piercy's comment is not entirely on point (you don't need to add a __init__.py since you're using a simple module) but they're right that autodoc will try to import the module and then inspect the content.

Hoever assuming your tree is

doc/conf.py
src/stack.py

then you're just adding the folder which contains your repository to the sys.path which is completely useless. What you need to do is add the src folder to sys.path, such that when sphinx tries to import stack it finds your module. So your line should be:

sys.path.insert(0, os.path.abspath('../src')

(the path should be relative to conf.py).

Of note: since you have something which is completely synthetic and should contain no secrets, an accessible repository or a zip file of the entire thing makes it much easier to diagnose issues and provide relevant help: the less has to be inferred, the less can be wrong in the answer.

Answer 4

IMHO running pip install --no-deps -e . in the top project folder (or where ever setup.py is) to get an "editable" install is a better alternative to get your package modules on the PYTHONPATH than altering it in docs/conf.py using sys.path.

Answer 5

For me installing the package via setup.py file and re-running corresponding commands fixed the problem:

$ python setup.py install