How to use setuptools packages and ext_modules with the same name?

Question

I got the following file structure for my Python C Extension project:

.
├── setup.py
├── source
    ├── cppimplementation
    │   └── fastfile.cpp
    └── fastfilepackage
        ├── __init__.py
        └── version.py

And I use the following setup.py file:

from setuptools import setup, Extension

setup(
        name= 'fastfilepackage',
        version= '0.1.1',
        package_dir = {
            '': 'source',
        },

        packages = [
            'fastfilepackage',
        ],

        ext_modules= [
            Extension(
                'fastfilepackage',
                [
                    'source/cppimplementation/fastfile.cpp',
                ]
            )
        ],
    )

I install them with:

$ pip3 --version
pip 19.1.1 (python 3.6)

$ python3 --version
Python 3.6.7

$ pip3 list
Package                Version      
---------------------- -------------
wheel                  0.33.1        
setuptools             40.8.0        
...

fastfilepackage$ pip3 install .

The problem is that when I install it, my Python C Extension module is overridden by fastfilepackage/version.py and fastfilepackage/__init__.py, i.e, after installing it, I got the following:

import fastfilepackage
print( dir( fastfilepackage ) )
# prints ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', 
#         '__name__', '__package__', '__path__', '__spec__', '__version__',
#         'version']

i.e., no FastFile class exported by source/cppimplementation/fastfile.cpp, but it has the fastfilepackage/version.py and fastfilepackage/__init__.py files.

This is the installed file structure:

.
└── dist-packages
    ├── fastfilepackage
    │   ├── __init__.py
    │   ├── __pycache__
    │   │   ├── __init__.cpython-36.pyc
    │   │   └── version.cpython-36.pyc
    │   └── version.py
    ├── fastfilepackage-0.1.1.dist-info
    │   ├── INSTALLER
    │   ├── LICENSE.txt
    │   ├── METADATA
    │   ├── RECORD
    │   ├── top_level.txt
    │   └── WHEEL
    └── fastfilepackage.cpython-36m-x86_64-linux-gnu.so

But if I remove the lines package_dir = { '': 'source', }, and packages = [ 'fastfilepackage', ], from my setup file, then, my Python C Extension module is correctly installed:

import fastfilepackage
print( dir( fastfilepackage ) )
# prints ['FastFile', '__doc__', '__file__', '__loader__', '__name__', 
#         '__package__', '__spec__']

i.e., it has the FastFile class exported by source/cppimplementation/fastfile.cpp, but it does not have the fastfilepackage/version.py and fastfilepackage/__init__.py files.

This was the installed file structure:

.
└── dist-packages
    ├── fastfilepackage-0.1.1.dist-info
    │   ├── INSTALLER
    │   ├── LICENSE.txt
    │   ├── METADATA
    │   ├── RECORD
    │   ├── top_level.txt
    │   └── WHEEL
    └── fastfilepackage.cpython-36m-x86_64-linux-gnu.so

How can I put ext_modules and packages to use the same package name under my setup.py without one overriding the other?

score 2 · Accepted Answer · answered May 21 '19 at 19:17

2

You cannot. The first one imported wins. You cannot have scripts/modules/packages/extensions with the same name — one overrides all others.

But you can to have one inside another. Make your extension named fastfilepackage.fastfilepackage and you can import fastfilepackage to import the Python package and import fastfilepackage.fastfilepackage to import the extension; or from fastfilepackage import fastfilepackage.

answered May 21 '19 at 19:17

phd

82,685
13
120
165

Can you give-me an example of `setup.py` doing this? – Evandro Coan May 21 '19 at 19:40
`Extension('fastfilepackage.fastfilepackage', ['source/cppimplementation/fastfile.cpp',])` – phd May 21 '19 at 20:10
1

I did this, but it did not work out of the box. I also had to edit `fastfilepackage/__init__.py` and manually include `from . import fastfilepackage` – Evandro Coan May 21 '19 at 20:15

Evandro Coan · Answer 2 · 2019-05-22T02:07:12.167

As a final solution, I completely removed all Python *.py code because they caused the C Extensions code to become 30% slower. Now my setup.py become like this:

from setuptools import setup, Extension

setup(
        name = 'fastfilepackage',
        version = '0.1.1',

        ext_modules = [
            Extension(
                name = 'fastfilepackage',
                sources = [
                    'source/fastfile.cpp',
                ],
                include_dirs = ['source'],
            )
        ],
    )

File structure:

.
├── setup.py
├── MANIFEST.in
├── README.md
├── LICENSE.txt
└── source
    ├── fastfile.cpp
    └── version.h

MANIFEST.in

include README.md
include LICENSE.txt

recursive-include source *.h

This is the installed file structure: (No *.py files anywhere = 100% performance)

.
└── dist-packages
    ├── fastfilepackage-0.1.1.dist-info
    │   ├── INSTALLER
    │   ├── LICENSE.txt
    │   ├── METADATA
    │   ├── RECORD
    │   ├── top_level.txt
    │   └── WHEEL
    └── fastfilepackage.cpython-36m-x86_64-linux-gnu.so

I just replaced the version.py directly by a C Extensions module attribute:

// https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue
const char* __version__ = "0.1.1";
PyObject_SetAttrString( thismodule, "__version__", Py_BuildValue( "s", __version__ ) );

References:

Define a global in a Python module from a C API
Original commit: Deprecated the Python implementation *.py

How to use setuptools packages and ext_modules with the same name?

2 Answers2

Linked