How to get autodoc to import packages that are locally written

Question

I have the following structure:

└──faulty_meter
    └── __init__.py
    └── run.py
    └── utils.py
    └── anomalous_kwh_detection
        └── __init__.py
        └── general_anomalies.py
        └── slowing_down.py
    └── docs
        └── Makefile
        └── build
        └── make.bat
        └── source
            └── conf.py
            └── index.rst

run.py imports utils.py and the files in anomalous_kwh_detection.

I run sphinx as follows from docs:

sphinx-quickstart
sphinx-apidoc -o ./source ..
make html

I get this warning: WARNING: autodoc: failed to import module 'run.py' from module 'faulty_meter'; the following exception was raised: No module named 'utils', which means autodoc can't find utils.py

I found a similar question here. I tried what was suggested, but could not really make it work.

In my case, what path should I include?

My conf.py file looks like this

import os
import sys
sys.path.insert(0, os.path.abspath('../../..'))
project = 'a'
copyright = '2021, b'
author = 'b'
release = '0.0.1'
extensions = ['sphinx.ext.autodoc','sphinx.ext.napoleon']
templates_path = ['_templates']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
html_theme = 'sphinx_rtd_theme'
html_static_path = ['_static']

and my index.rst looks like this:

.. a documentation master file, created by
   sphinx-quickstart on Thu Aug 19 21:54:31 2021.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to a's documentation!
=============================

.. toctree::
   :maxdepth: 6
   :caption: Contents:

   modules

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

Everything works nicely, except that run.py is ignored as indicated by the warning. How to fix that?

Edit: I attach an example of the generated documentation. It generates the documentation for utils correctly, but ignores run (it actually is run_anom_detection, but the name does not matter).

image

2nd Edit:

I'll provide a minimal reproducing example:

Let's have the following folder structure:

└──faulty_meter
    └── __init__.py
    └── run.py
    └── utils.py
    └── anomalous_kwh_detection
        └── __init__.py
        └── general_anomalies.py
    └── docs
        └── Makefile
        └── build
        └── make.bat
        └── source
            └── conf.py
            └── index.rst

Then utils.py can look like this

def test1(x, y):
    """
    This is just a quick test

    Parameters
    ----------
    x: array
       random array
    y: array
       another random array
    """
    return x+y

general_anomaly.py can look like this

def test2(x, y):
    """
    This is just a quick test

    Parameters
    ----------
    x: array
       random array
    y: array
       another random array
    """
    return x-y

and run.py can be

import utils
from anomalous_kwh_detection import general_anomalies as ga

def test_run(x, y, z):
    """
    This is the function that is not documented
    """
    p = utils.test1(x, y)+ ga.test2(y, z)
    return p

When I'm trying to debug problems like these I sometimes cd docs/source and then start an interactive python session. Then you can try a lot of different sys.path modifications and imports until you've confirmed you've set your sys.path correctly to find run.py from the dir where conf.py will run. — morric, Aug 20 '21 at 20:08
@morric I suspect I need to add something else because it generates documentation for ```utils```, so it definitely has access to ```utils```. and I have tried all paths that I think to be reasonable, but the problem persists. However, could you be a little more specific about your suggestion? Thanks! — Schach21, Aug 20 '21 at 20:21
@Schach21, that's interesting - and suggests your sys.path is pointing to the right dir. The value in trying to import from docs/source in an interactive python session is you can use the os.sys.path module to adjust your sys.path for the session, and then try to import run and see what error you get. And you can iterate quickly. You're also isolating the import from the rest of the apidoc functionality. EDIT: I see now that Sphinx was getting to run.py - just not from run.py to utils.py, so I'm not sure if this approach will help much. — morric, Aug 20 '21 at 20:28
I'm just wondering, if Sphinx can find utils.py and run.py ok, it's doing all it can. So maybe double-check: can run.py find utils.py? — morric, Aug 20 '21 at 20:34
What do you mean if ```run.py``` can find ```utils.py```? I can execute ```run.py``` and it generates no error. The python code works perfectly. I attached a picture of what I see. And run is actually ```run_anom_detection``` but that does not change anything. — Schach21, Aug 20 '21 at 20:49
I have added a minimal reproducible example. I did not include my ```conf.py``` or ```index.rst``` because I had included them before. — Schach21, Aug 21 '21 at 15:09

score 1 · Accepted Answer · answered Aug 21 '21 at 18:54

The issue here is the interpretation of import statements in Python packages. In run.py, you have:

import utils

This is an implicit relative import, which is not supported in Python 3.

It works if you change to either an explicit relative import,

from . import utils

or an absolute import,

import faulty_meter.utils as utils

The other statement in run.py will work if a dot is added:

from .anomalous_kwh_detection import general_anomalies as ga

Note: When python run.py is run from the command line, the code is not executed as part of a package and the imports specified above do not work. Instead you can use python -m faulty_meter.run.

References:

Thanks! It works like a charm. For future readers: it is not necessary to separate ```build``` and ```source``` when using ```sphinx-quickstart```. My problem was the incorrect importing, as pointed out in the answer. — Schach21, Aug 21 '21 at 19:43

How to get autodoc to import packages that are locally written

1 Answers1