Calling script in standard project directory structure (Python path for bin subdirectory)

Question

I am experimenting with putting my Python code into the standard directory structure used for deployment with setup.py and maybe PyPI. for a Python library called mylib it would be something like this:

mylibsrc/
  README.rst
  setup.py
  bin/
    some_script.py
  mylib/
    __init.py__
    foo.py

There's often also a test/ subdirectory but I haven't tried writing unit tests yet. The recommendation to have scripts in a bin/ subdirectory can be found in the official Python packaging documentation.

Of course, the scripts start with code that looks like this:

#!/usr/bin/env python
from mylib.foo import something
something("bar")

This works well when it eventually comes to deploying the script (e.g. to devpi) and then installing it with pip. But if I run the script directly from the source directory, as I would while developing new changes to the library/script, I get this error:

ImportError: No module named 'mylib'

This is true even if the current working directory is the root mylibsrc/ and I ran the script by typing ./bin/some_script.py. This is because Python starts searching for packages in the directory of the script being run (i.e. from bin/), not the current working directory.

What is a good, permament way to make it easy to run scripts while developing packages?

Here is a relevant other question (especially comments to the first answer).

The solutions for this that I've found so far fall into three categories, but none of them are ideal:

1. Manually fix up your Python's module search path somehow before running your scripts.
   • You can manually add mylibsrc to my PYTHONPATH environment variable. This seems to be the most official (Pythonic?) solution, but means that every time I check out a project I have to remember to manually change my environment before I can run any code in it.
   • Add . to the start of my PYTHONPATH environment variable. As I understand it this could have some security problems. This would actually be my favoured trick if I was the only person to use my code, but I'm not, and I don't want to ask others to do this.
   • While looking at answers on the internet, for files in a test/ directory I've seen recommendations that they all (indirectly) include a line of code sys.path.insert(0, os.path.abspath('..')) (e.g. in structuring your project). Yuck! This seems like a bearable hack for files that are only for testing, but not those that will be installed with the package.
   • Edit: I have since found an alternative, which turns out to be in this category: by running the scripts with Python's -m script, the search path starts in the working directory instead of the bin/ directory. See my answer below for more details.
2. Install the package to a virtual environment before using it, using a setup.py (either running it directly or using pip).
   • This seems like overkill if I'm just testing a change that I'm not sure is even syntactically correct yet. Some of the projects I'm working on aren't even meant to be installed as packages but I want to use the same directory structure for everything, and this would mean writing a setup.py just so I could test them!
   • Edit: Two interesting variants of this are discussed in the answers below: the setup.py develop command in logc's answer and pip install -e in mine. They avoid having to re-"install" for every little edit, but you still need to create a setup.py for packages you never intend to fully install, and doesn't work very well with PyCharm (which has a menu entry to run the develop command but no easy way to run the scripts that it copies to the virtual environment).
3. Move the scripts to the project's root directory (i.e. in mylibsrc/ instead of mylibsrc/bin/).
   • Yuck! This is a last resort but, unfortunately, this seems like the only feasible option at the moment.

Answer 1

Run modules as scripts

Since I posted this question, I've learnt that you can run a module as if it were a script using Python's -m command-line switch (which I had thought only applied to packages).

So I think the best solution is this:

• Instead of writing wrapper scripts in a bin subdirectory, put the bulk of the logic in modules (as you should anyway), and put at the end of relevant modules if __name__ == "__main__": main(), as you would in a script.
• To run the scripts on the command line, call the modules directly like this: python -m pkg_name.module_name
• If you have a setup.py, as Alik said you can generate wrapper scripts at installation time so your users don't need to run them in this funny way.

PyCharm doesn't support running modules in this way (see this request). However, you can just run modules (and also scripts in bin) like normal because PyCharm automatically adds the project root to the PYTHONPATH, so import statements resolve without any further effort. There are a few gotchas for this though:

• The main problem the working directory will be incorrect, so opening data files won't work. Unfortunately there is no quick fix; the first time you run each script, you must then stop it and change its configured working directory (see this link).
• If your package directory is not directly within the root project directory, you need to mark its parent directory as a source directory in the project structure settings page.
• Relative imports don't work i.e. you can do from pkg_name.other_module import fn but not from .other_module import fn. Relative imports are usually poor style anyway, but they're useful for unit tests.
• If a module has a circular dependency and you run it directly, it will end up being imported twice (once as pkg_name.module_name and once as __main__). But you shouldn't have circular dependencies anyway.

Bonus command line fun:

• If you still want to put some scripts in bin/ you can call them with python -m bin.scriptname (but in Python 2 you'll need to put an __init__.py in the bin directory).
• You can even run the overall package, if it has a __main__.py, like this: python -m pkg_name

Pip editable mode

There is an alternative for the command line, which is not as simple, but still worth knowing about:

• Use pip's editable mode, documented here
• To use it, make a setup.py, and use the following command to install the package into your virtual environment: pip install -e .
• Note the trailing dot, which refers to the current directory.
• This puts the scripts generated from your setup.py in your virtual environment's bin directory, and links to your package source code so you can edit and debug it without re-running pip.
• When you're done, you can run pip uninstall pkg_name
• This is similar to setup.py's develop command, but uninstallation seems to work better.

Answer 2

The simplest way is to use setuptools in your setup.py script, and use the entry_points keyword, see the documentation of Automatic Script Creation.

In more detail: you create a setup.py that looks like this

from setuptools import setup

setup(
    # other arguments here...
    entry_points={
        'console_scripts': [
            'foo = my_package.some_module:main_func',
            'bar = other_module:some_func',
        ],
        'gui_scripts': [
            'baz = my_package_gui:start_func',
        ]
    }
)

then create other Python packages and modules underneath the directory where this setup.py exists, e.g. following the above example:

.
├── my_package
│   ├── __init__.py
│   └── some_module.py
├── my_package_gui
│   └── __init__.py
├── other_module.py
└── setup.py

and then run

$ python setup.py install

or

$ python setup.py develop

Either way, new Python scripts (executable scripts without the .py suffix) are created for you that point to the entry points you have described in setup.py. Usually, they are at the Python interpreter's notion of "directory where executable binaries should be", which is usually on your PATH already. If you are using a virtual env, then virtualenv tricks the Python interpreter into thinking this directory is bin/ under wherever you have defined that the virtualenv should be. Following the example above, in a virtualenv, running the previous commands should result in:

bin
├── bar
├── baz
└── foo