103

I am trying to make a python package which I want to install using pip install . locally. The package name is listed in pip freeze but import <package> results in an error No module named <package>. Also the site-packages folder does only contain a dist-info folder. find_packages() is able to find packages. What am I missing?

import io
import os
import sys
from shutil import rmtree

from setuptools import find_packages, setup, Command

# Package meta-data.
NAME = '<package>'
DESCRIPTION = 'description'
URL = ''
EMAIL = 'email'
AUTHOR = 'name'

# What packages are required for this module to be executed?
REQUIRED = [
    # 'requests', 'maya', 'records',
]

# The rest you shouldn't have to touch too much :)
# ------------------------------------------------
# Except, perhaps the License and Trove Classifiers!
# If you do change the License, remember to change the Trove Classifier for that!

here = os.path.abspath(os.path.dirname(__file__))



# Where the magic happens:
setup(
    name=NAME,
    #version=about['__version__'],
    description=DESCRIPTION,
    # long_description=long_description,
    author=AUTHOR,
    author_email=EMAIL,
    url=URL,
    packages=find_packages(),
    # If your package is a single module, use this instead of 'packages':
    # py_modules=['mypackage'],

    # entry_points={
    #     'console_scripts': ['mycli=mymodule:cli'],
    # },
    install_requires=REQUIRED,
    include_package_data=True,
    license='MIT',
    classifiers=[
        # Trove classifiers
        # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python',
        'Programming Language :: Python :: 2.6',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
        'Programming Language :: Python :: Implementation :: CPython',
        'Programming Language :: Python :: Implementation :: PyPy'
    ],

)
hoefling
  • 59,418
  • 12
  • 147
  • 194
André Betz
  • 1,291
  • 2
  • 10
  • 9
  • 7
    Run `pip uninstall -y pkgname && pip install -v .` and check the build log whether the source files are processed (look for the lines containing `copying file ` and `adding `). If the files were added, run `pip show -f pkgname` and add the output of both commands to the question. – hoefling May 29 '18 at 13:34
  • 1
    Thanks a lot. That made me realize that I gave a different package name to the `setup.py` than the name of the directory. – André Betz May 29 '18 at 13:48
  • I had the same problem until I realized it was a stupid mistake: I started the wheel building process while the project was still copying to a new location. That way, it did not include any python code. All attempts at installing the wheel with different settings were bound to fail because the wheel did not in fact contain any usable code files to be installed. – Cerno Aug 14 '18 at 08:54
  • hoefling, could you type up a more complete answer based on your comment? I don't know how to check the build log and I'm not sure what you want us to add to `pip show` – Jake Stevens-Haas Jan 25 '19 at 00:37

5 Answers5

145

Since the question has become quite popular, here are the diagnosis steps to go through when you're missing files after installation. Imagine having an example project with the following structure:

root
├── spam
│   ├── __init__.py
│   ├── data.txt
│   ├── eggs.py
│   └── fizz
│       ├── __init__.py
│       └── buzz.py
├── bacon.py
└── setup.py

Now I run pip install ., check that the package is installed:

$ pip list
Package    Version
---------- -------
mypkg      0.1    
pip        19.0.1 
setuptools 40.6.3 
wheel      0.32.3

but see neither spam, nor spam/eggs.py nor bacon.py nor spam/fizz/buzz.py in the list of files belonging to the installed package:

$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
  mypkg-0.1.dist-info/DESCRIPTION.rst
  mypkg-0.1.dist-info/INSTALLER
  mypkg-0.1.dist-info/METADATA
  mypkg-0.1.dist-info/RECORD
  mypkg-0.1.dist-info/WHEEL
  mypkg-0.1.dist-info/metadata.json
  mypkg-0.1.dist-info/top_level.txt

So what to do now?

Diagnose by inspecting the wheel build log

Unless told not to do so, pip will always try to build a wheel file and install your package from it. We can inspect the log for the wheel build process if reinstalling in the verbose mode. First step is to uninstall the package:

$ pip uninstall -y mypkg
...

then install it again, but now with an additional argument:

$ pip install . -vvv
...

Now if I inspect the log:

$ pip install . -vvv | grep 'adding'
  adding 'mypkg-0.1.dist-info/METADATA'
  adding 'mypkg-0.1.dist-info/WHEEL'
  adding 'mypkg-0.1.dist-info/top_level.txt'
  adding 'mypkg-0.1.dist-info/RECORD'

I notice that no files from the spam directory or bacon.py are mentioned anywhere. This means they were simply not included in the wheel file and hence not installed by pip. The most common error sources are:

Missing packages: check the packages argument

Verify you have passed the packages argument to the setup function. Check that you have mentioned all of the packages that should be installed. Subpackages will not be collected automatically if only the parent package is mentioned! For example, in the setup script

from setuptools import setup

setup(
    name='mypkg',
    version='0.1',
    packages=['spam']
)

spam will be installed, but not spam.fizz because it is a package itself and must be mentioned explicitly. Fixing it:

from setuptools import setup

setup(
    name='mypkg',
    version='0.1',
    packages=['spam', 'spam.fizz']
)

If you have lots of packages, use setuptools.find_packages to automate the process:

from setuptools import find_packages, setup

setup(
    name='mypkg',
    version='0.1',
    packages=find_packages()  # will return a list ['spam', 'spam.fizz']
)

In case you are missing a module:

Missing modules: check the py_modules argument

In the above examples, I will be missing bacon.py after installation since it doesn't belong to any package. I have to provide its module name in the separate argument py_modules:

from setuptools import find_packages, setup

setup(
    name='mypkg',
    version='0.1',
    packages=find_packages(),
    py_modules=['bacon']
)

Missing data files: check the package_data argument

I have all the source code files in place now, but the data.txt file is still not installed. Data files located under package directories should be added via the package_data argument. Fixing the above setup script:

from setuptools import find_packages, setup

setup(
    name='mypkg',
    version='0.1',
    packages=find_packages(),
    package_data={'spam': ['data.txt']},
    py_modules=['bacon']
)

Don't be tempted to use the data_files argument. Place the data files under a package and configure package_data instead.

After fixing the setup script, verify the package files are in place after installation

If I now reinstall the package, I will notice all of the files are added to the wheel:

$ pip install . -vvv | grep 'adding'
  adding 'bacon.py'
  adding 'spam/__init__.py'
  adding 'spam/data.txt'
  adding 'spam/eggs.py'
  adding 'spam/fizz/__init__.py'
  adding 'spam/fizz/buzz.py'
  adding 'mypkg-0.1.dist-info/METADATA'
  adding 'mypkg-0.1.dist-info/WHEEL'
  adding 'mypkg-0.1.dist-info/top_level.txt'
  adding 'mypkg-0.1.dist-info/RECORD'

They will also be visible in the list of files belonging to mypkg:

$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
  __pycache__/bacon.cpython-36.pyc
  bacon.py
  mypkg-0.1.dist-info/INSTALLER
  mypkg-0.1.dist-info/METADATA
  mypkg-0.1.dist-info/RECORD
  mypkg-0.1.dist-info/WHEEL
  mypkg-0.1.dist-info/top_level.txt
  spam/__init__.py
  spam/__pycache__/__init__.cpython-36.pyc
  spam/__pycache__/eggs.cpython-36.pyc
  spam/data.txt
  spam/eggs.py
  spam/fizz/__init__.py
  spam/fizz/__pycache__/__init__.cpython-36.pyc
  spam/fizz/__pycache__/buzz.cpython-36.pyc
  spam/fizz/buzz.py
hoefling
  • 59,418
  • 12
  • 147
  • 194
  • On Linux, I've done all the above. The install works fine, but from an IPython notebook `import ` fails, but `!pip freeze | grep ` lists it. Is there anything else that needs to be done before packages can be imported? – user1717828 Jul 07 '19 at 19:22
  • @user1717828 most probably you have multiple Python versions installed and the default `pip` command is aliased to `pip` from the wrong version of Python. Run `!pip -V` and smth like `import sys; print(sys.version); print(sys.executable)` from IPython to check the versions. The rest highly depends on Python version and Linux distro, but usually trying available aliases like `pip2` or `pip3` or `pip3.6` should lead you to the installation for the correct version. – hoefling Jul 07 '19 at 20:05
  • I just checked those things, and I don't think that's it. Instead, I'm wondering if I'm just not importing correctly. I was trying to do `import mypkg`, but maybe I should be doing `import spam` or `import spam.fizz`? Sorry if this seems daft, but perhaps you could augment the answer with the final step of how to actually use the installed package. – user1717828 Jul 08 '19 at 00:04
  • 1
    I should add that I am curious why `import mypkg` fails. – user1717828 Jul 08 '19 at 00:04
  • 3
    Are you trying to import the distribution name (what is put under `name` argument in the setup script) or one of the package names (what is put under `packages` argument)? You can't import the distribution name. Overall, that's an issue unrelated to this one and I'd suggest to ask another question instead. – hoefling Jul 08 '19 at 09:02
  • 5
    Excellent answer! Thanks. – f10w Mar 28 '20 at 22:04
  • 4
    Another common source of error: Every package must have an `__init__.py` file for setuptools to find it. It's easy to forget this if you're used to working with development installs (`-e`), which DON'T require this. – Tristan Brown Sep 18 '20 at 17:40
  • @TristanBrown I think your comment deserves to be a separate answer. – hoefling Sep 22 '20 at 14:57
  • @hoefling Thank you for taking time and writing this solution. – tush4r Feb 11 '21 at 14:42
  • @hoefling `pip install . -vvv | grep 'adding'` - diagnostic gold, thank you! – jtlz2 Aug 04 '21 at 07:18
  • @hoefling This only works for me with `*.py` files - not with `*.md` or anything else - what could I be doing wrong? :\ – jtlz2 Aug 04 '21 at 07:32
  • @jtlz2 make sure you've added the `*.md` glob for the package dir containing the markdown file in `package_data` - there's a mentioning of it in the answer. – hoefling Aug 04 '21 at 08:01
  • @hoefling I am still stumped in getting it working. Is there chat here where you might be able to help me if you are willing and able? I am loathe to add a question for my specific case.. – jtlz2 Aug 04 '21 at 08:31
  • @hoefling I fixed it by removing my `py_modules=` - I am not sure under what circumstances I should use that, why removing it fixed it, or indeed what the whole point of it is... – jtlz2 Aug 04 '21 at 10:31
  • What is the alternative for `pip install . -vvv | grep 'adding'` in windows? – Muhammad Yasirroni Dec 27 '21 at 05:30
  • 1
    @hoefling Thanks a lot for the detailed answer. You taught me how to fish ;) – eidal May 25 '22 at 09:34
  • Thank you for this answer, got me out of a hole after a lot of searching – ljdyer Nov 05 '22 at 22:22
2

Make certain that your src files are in example_package_YOUR_USERNAME_HERE (this is the example package name that is used in the docs) and not in src. Errantly putting the files in src can have the effect described in the question.

Reference: https://packaging.python.org/en/latest/tutorials/packaging-projects/

The package should be set up like this:

packaging_tutorial/
└── src/
    └── example_package_YOUR_USERNAME_HERE/
        ├── __init__.py
        └── example.py
stackhatter
  • 304
  • 2
  • 11
1

For me, I noticed something weird if you do this:

# Not in the setup.py directory
python /path/to/folder/setup.py bdist_wheel

It will only install the .dist-info folder in your site-packages folder when you install the wheel.

However, if you do this:

cd /path/to/folder \
&& python setup.py bdist_wheel

The wheel will include all your files.

jmcgrath207
  • 1,317
  • 2
  • 19
  • 31
0

If you are on Windows 10+, one way you could make sure that you had all the correct installations was to click start in the bottom left-hand corner and search cmd.exe and right-click on "Command Prompt" (Make sure you choose "Run as Administrator"). Type "cd path to your Python 3.X installation". You can find this path in File Explorer (go to the folder where Python is installed) and then at the top. Copy this, and put it in where I wrote above path to your Python 3.X installation. Once you do that and click enter, type "python -m pip install package" (package signifies the package you would like to install). Your Python program should now work perfectly.

Arnav Poddar
  • 354
  • 2
  • 18
0

I had the same problem, and updating setuptools helped:

python3 -m pip install --upgrade pip setuptools wheel

After that, reinstall the package, and it should work fine :)

Ivan Konovalov
  • 39
  • 3