Python: Generate function stubs from C module

Question

I made a Python module in C/C++ with Python C API. I use setuptools.Extension in my setup.py.

It creates one .py file which loads a python module from some compiled .pyd file:

def __bootstrap__():
    global __bootstrap__, __loader__, __file__
    import sys, pkg_resources, imp
    __file__ = pkg_resources.resource_filename(__name__, 'zroya.cp36-win32.pyd')
    __loader__ = None; del __bootstrap__, __loader__
    imp.load_dynamic(__name__,__file__)
__bootstrap__()

But it does not generate python stubs for IDE autocomplete feature. I would like all exported functions and classes to be visible from .py file:

def myfunction_stub(*args, **kwargs):
    """
    ... function docstring
    """
    pass

Is it possible? Or do I have to create some python "preprocessor" which loads data from .pyd file and generate stubs with docstrings?

Source code is available on github.

What is the bootstrap.py stub for? Python extension modules can be `import`ed directly in python and their docstrings are available as normal. There does not seem to be any benefit to manually importing a specific file. — danny, Mar 21 '18 at 17:13
@DavidW, When my C code is compiled, there are zroya.pyd and zroya.py files generated (zroya is name of my python extension). First one contains compiled code, the later one defines __bootstrap__ function. But there are no stubs for module functions nor classes. — Lorin, Mar 22 '18 at 09:43

score 2 · Answer 1 · answered Sep 16 '22 at 12:00

This question is old but since it didn't contain an answer as I was looking into this issue I thought I'd provide what worked in my case.

I had a python module developed using the c-api with the following structure:

my_package/
├── docs
│   └── source
├── package_libs
│   ├── linux
│   └── win
│       ├── amd64
│       └── i386
├── package_src
│   ├── include
│   └── source
└── tests

the typegen command of the mypy package can generate stubs for packages pretty well. The steps used were to first compile the package as you normally would with your existing setup.py for example. Then generated the stubs for the generated .pyd or .so. In my case the easiest was to install the whole package using pip for example and then calling stubgen on the whole module e.g:

pip install my_package
pip install mypy
stubgen my_package

This generates a my_package.pyi file which can then be included in the package data of your setup.py file as follows:

.
.
.
setup(
      .
      .
      .
        package=["my_package"],
        package_data={"my_package": ["py.typed", "my_package.pyi", "__init__.pyi"]},
      .
      .
      .
)

.
.
.

In there I include an empty py.typed file to let utilities know that the package contains type stubs, the generated my_package.pyi file and an __init__.pyi file containing only the import of the stubs to make them available at the top level of my package as they are in module.

from my_package import *

This works for me and is reproducible even in CI environments where we generate the stubs before publishing the package so that they don't need to be manually updated or checked for discrepancy.

The final source repository looks like this with the added files :

my_package/
├── docs
│   └── source
├── my_package
│   ├── __init__.pyi
│   ├── my_package.pyi # generated by stubgen upon successful CI build in my case
│   └── py.typed
├── package_libs
│   ├── linux
│   └── win
│       ├── amd64
│       └── i386
├── package_src
│   ├── include
│   └── source
└── tests

Thanks, this helped a lot! I had to do `stubgen -p my_package` though. However, using stubgen does not include the docstring and therefore there are no parameter hints and such :( — Thomas Wagenaar, Oct 22 '22 at 17:19

score 2 · Answer 2 · answered Oct 22 '22 at 17:58

Unfortunately, mypy's stubgen does not (yet) include docstrings and signatures. However, it is relatively easy to automatically generate your own stub's using the Python native inspect package. For example, I use something along the lines of:

import my_package
import inspect

with open('my_package.pyi', 'w') as f:
    for name, obj in inspect.getmembers(nf):
        if inspect.isclass(obj):
            f.write('\n')
            f.write(f'class {name}:\n')

            for func_name, func in inspect.getmembers(obj):
                if not func_name.startswith('__'):
                    try:
                        f.write(f'    def {func_name} {inspect.signature(func)}:\n')
                    except:
                        f.write(f'    def {func_name} (self, *args, **kwargs):\n')
                    f.write(f"      '''{func.__doc__}'''")
                    f.write('\n    ...\n')

So essentially, first install your package and afterwards you can run this script to create a .pyi. You can change it easily to your liking! Note that you must correctly define docstrings in your C/C++ code: https://stackoverflow.com/a/41245451/4576519

Python: Generate function stubs from C module

2 Answers2

Linked