How to locally develop a python package?

Question

I'm trying to make changes to an existing python module, and then test it locally. What's the best way to do this?

I cloned the github module and made changes, but I'm not sure how to import the local package instead of the already installed one.

Answer 1

The easiest way to do such testing would be to create a virtual environment, and then installing the package in development mode.

Assuming you are on Linux it would look something like this.

$ virtualenv dev_env
$ source dev_env/bin/activate
$ cd ~/project_folder
$ pip install -e .

This workflow would not overwrite the already installed package on your system. Another maybe simpler alternatives would to just use an IDE that handles most of this for you, e.g. PyCharm.

Answer 2

TL; DR

You can:

• Overwrite a module/package by creating another one with the same name in the same folder as the script you'll run.
• Use the develop mode.

I recommend reading this article that explains pretty well modules and packages.

---

Overwriting the module/package

Description

You need to create a module or a package (it doesn't make difference) using the same name as the module/package you want and put it in the same folder as the script that it's going to use it.

This because modules are searched starting from the sys.path variable (where the first element is the script's directory)

Example

1. Create a script with the following contents:

import platform

print(platform.system())

2. Launching it (python your_test_script.py) should return:

   

3. Now in the same directory of the previous test script create a file named exactly platform.py with the following contents:

def system():
    """Just a docstring passing by"""
    return "We have just overwritten default 'platform' module...\nFeel the force!"

4. If you launch the script now, you'll notice the output is different:

   

---

Develop mode

Description

Better option if your project is more complicated.

From the root of your package (where you'd launch the build):

pip install -e ./ 

Now you're able to edit code and see the changes in real time..

---

From The Joy of Packaging:

It puts a link (actually *.pth files) into the python installation to your code, so that your package is installed, but any changes will immediately take effect.

This way all your test code, and client code, etc, can all import your package the usual way.

No sys.path hacking

Answer 3

One way consists in using sys.path().
For example:

import sys
sys.path.insert(0, path/to/module)

In this way, you give priority to a specific path when looking for a module.
This means that the module you want to import will be searched first in path/to/module and after in the other directories already in sys.path.

The advantage of this approach is that this new order will hold only inside your script without changing the import order of the other ones.

Note: For development purposes you should use a virtualenv as suggested by @eandersson.

Answer 4

You should probably be doing most of your development work in a virtual environment. Your workflow for this could look like:

# activate the virtual environment in ~/vpy
. $HOME/vpy/bin/activate

# install my app and its dependencies
cd $HOME/src/myapp
pip install -e .

# use my forked library instead
cd $HOME/src/forkedlib
pip install -e .
pytest # or whatever tests the forked lib has

# try it out with my application too
cd $HOME/src/myapp
pytest # or whatever tests your app has
myapp

pip install -e does some magic so that, whenever you import the module in the library, it gets routed directly to the checked-out source tree, so if you make edits in forkedlib and then re-run myapp, you'll see those changes directly.

When you're done, you can pip uninstall forkedlib and then re-run pip install -e . to reinstall your application's (declared) dependencies. (Or delete and recreate the virtual environment, if that's easier.)

Answer 5

The approach of the answer by abc adding the module path to the system path is fine for local, instant testing, but it's not the full solution, for example when C code needs to be compiled or command line hooks must be set. For a full test you might want to install the package instead.

A typical Python has a setup.py and can be packaged into a distribution file (wheel, ...) which can then be installed locally from a local file.

The workflow would be:

• create a package distributable (may include a command like python setup.py bdist_wheel)
• create a new virtual environment for testing (or de-install any previously installed, non-modified version of the package)
• installe the package from the locally created distributable (may be as simple as pip install --no-index --find-links=..)
• run the tests

This results in exactly the same situation every future user of the package will find itself in and is a complete test (including the installing process), but it's also a lot of effort, that's why I usually only us the system path method during development, but the full installation way only directly before a release.