5

The situation

You have 2 software products in development, a library which presents an API and a GUI tool that exposes the library for end users. Additionally, you expect a lot of technical staff at your place to use the library as a building block for all sorts of relevant custom code, tools and assets.

Both software products (library and GUI tool) are actively in development and influence each other. For both you want the most easy way of distribution and devenv setup, using pip:

pip install gui_tool
or
pip install library

Gui Tool (use case 1)

The installation of the GUI tool happens via pip and the dependencies are noted with a hard version number within the setup.py. Your library is one of those dependencies:

...
install_requires = ['library==1.2, package_x==0.3, package_y==0.6'],
...

The installation procedure consists of installing the tools and resolving the dependencies into a fresh virtualenv. Because every dependency version is hardwired, you control a stable and consistent installation. Nightly builds/bleeding edge dependencies can be controlled by devs through manually updating library to newer versions:

pip install --upgrade library  # get the latest nightly build/hotfix release on your own

Custom code/tools (use case 2)

As mentioned, a lot of code and custom tools might be built using the API that your library provides. Everybody willing to use it should install/update it through pip easily with the oneliner from the top.

Problem

Fellow GUI tool developers should be able to pull in nightly builds/hotfix releases of the library dependency with pip. Other staff, using library as a building block somewhere, should always get the latest stable version using pip. You want to remain one unique release procedure for library providing stable versions, bleeding edge and hotfix releases through X.Y.Z versioning.

There are a few possible solutions to this, like:

  • maintaining a readme for the custom users that states the stable versions they should be installing with pip
  • or setting up some magic within the setup.py for the Gui Tool cloning the git repo and using it through python setup.py develop (Versioning can then be handled by devs via checkouts in the repo).

However, none of these seem particularly elegant, so I am interested in your solutions, ideas or best practices for stable/bleeding edge/nightly build dependency management for Python?

timmwagener
  • 2,368
  • 2
  • 19
  • 27

2 Answers2

2

Managing a "bleeding edge" version in pip can be achieved by using the --pre flag of pip

From pip install --help:

--pre Include pre-release and development versions. By default, pip only finds stable versions.

You should add classifiers to your project, in particular set your stable release to Development Status :: 5 - Production/Stable and your "bleeding edge releases" to anything below 5.

This is how major python packages manage their alphas, e.g.: the django project with currently 1.9a in alpha and 1.8.5 in stable.

To upgrade to the latest version flagged as Development Status :: 3 - Alpha :

pip install --pre --upgrade library

Users using the library as building blocks need not now about the alpha releases and will go with a regular pip install library which will install the latest stable flagged version.

Sebastian Wozny
  • 16,943
  • 7
  • 52
  • 69
0

I am not sure if you need to manage on library or several libraries?

It seems like you have a case of complex Python dependency tracking and deployment. There are few large Python projects which face the same issue with the need to track several different release channels, betas and bleeding edge. Most notable them is Plone which sports more than 300 MB of Python egg source code.

Plone used a buildout as the dependency manager instead of pip to tackle the complexity. Buildout offers mix and match configuration files for Python dependencies.

To quickly grasp how this happens see Plone core buildouts

Warning Buildout is arcane and rough on the edges like just surfaced volcano island of hot lava. Is it more elegant than generating pip requirements.txt with custom scripts? Yes if you have more than one library which needs to be managed.

Mikko Ohtamaa
  • 82,057
  • 50
  • 264
  • 435