8

Up to now we installed psutil via our custom pypi server.

Now we have a new environment where we should not install gcc.

Now the old way (pip starts gcc during install) does not work any more.

The context:

  • Linux servers
  • python inside virtualenv
  • All code needs to be deployed from our data center (without internet access)

I see these alternatives:

RPM

Create a RPM. Since we already run our virtualenv with --system-site-packages this works. This forces all virtualenvs on the server to use the same version of psutil. But this would be no big problem.

Wheel

I have never used that.

Freeze

Use a tool like cx_freeze. I have never done this before.

Other solution?

I guess there are other, maybe better, ways to solve this.

Background

psutil is just an example in this case. The same question comes up for other python packages containing c-extensions. Imagine there are no RPMs here yet.

guettli
  • 25,042
  • 81
  • 346
  • 663
  • I'm pretty sure I'd just use a wheel for this situation. You then have to build a wheel per target architecture (a good example of this is Gohlke's page of [wheels for Windows](http://www.lfd.uci.edu/~gohlke/pythonlibs/) - a compelling reason for doing this is that most Windows users don't have the correct setup to compile C for extensions). Given the bounty you've offered - is there anything in particular that makes you think compiling wheels and then using e.g. `pip install` on the wheel (`.whl`) on the target machine might be problematic for you? – J Richard Snape May 27 '16 at 10:15
  • Looks like this could be a dupe of http://stackoverflow.com/questions/31380578/how-to-avoid-building-c-library-with-my-python-package – Peter Brittain May 27 '16 at 21:39

2 Answers2

7

The most idiomatic way is to use wheels. In fact, your use case is one of the reasons why the wheel format was created.

Building a platform wheel is easy:

python setup.py bdist_wheel

You might get an error "invalid command 'bdist_wheel'". In this case you have to install the wheel package:

pip install wheel

After building the wheel, it is in e.g. dist/psutil-4.2.0-cp27-cp27mu-linux_x86_64.whl. You can install it by:

pip install dist/psutil-4.2.0-cp27-cp27mu-linux_x86_64.whl

In general, installing and using the wheel only works on a system which is binary-compatible. As this is not guaranteed across different Linux distributions and versions, there are restrictions when uploading wheels to the central PyPI. These restrictions don't apply when running your own PyPI server.

You can upload the wheel to your custom PyPI by:

python setup.py bdist_wheel upload --repository <url-to-custom-pypi>

And install it from your custom PyPI by e.g.:

pip install --index-url <url-to-custom-pypi> psutil
Dave
  • 7,555
  • 8
  • 46
  • 88
Manuel Jacob
  • 1,897
  • 10
  • 21
  • Wheel is the way to go for sure! – Philippe Ombredanne May 29 '16 at 12:06
  • I do not think that there are any restriction to upload linux-specific wheels to Pypi? Do you have a link/pointer to this? Also there is a new support for a "many-linux" platform tag and wheel build – Philippe Ombredanne May 29 '16 at 12:08
  • It's according to the "Python Packaging User Guide" (https://packaging.python.org/en/latest/distributing/#platform-wheels), which "aims to be the authoritative resource on how to package, publish and install Python distributions using current tools". In the answer I wrote "according to the documentation", which might not be 100% correct technically. I'll check both and update my answer accordingly. – Manuel Jacob May 29 '16 at 12:49
  • The guide is out of date with the latest support for PEP513 and "manylinux" tags. See https://github.com/pypa/manylinux > The goal of the manylinux project is to provide a convenient way to distribute binary Python extensions as wheels on Linux. Wheel packages compliant with those tags can be uploaded to PyPI (for instance with twine) and can be installed with pip 8.1 and later. – Philippe Ombredanne May 29 '16 at 12:59
  • Indeed wheels conforming to the "manylinux" policy can be uploaded to the central PyPI (https://www.python.org/dev/peps/pep-0513/). I asked the Python Packaging User Guide authors to update the documentation. This isn't relevant to the question however because the question author uses a custom PyPI, where this restriction doesn't apply. EDIT: you answered a bit faster. ;) – Manuel Jacob May 29 '16 at 13:07
2

There are disadvantages in the package approach (rpm, deb, wheel, etc):

  • The C libraries binaries, potentially needed, are not portable. So that implies that it can fail if the environment has incompatible libraries
  • You need to work out to keep the QA environment and the production environment exactly the same, with the same lib versions. Otherwise you may have problems in production that wouldn't be caught in QA time. I had this problem, for example, with a version of the MySQL driver panicking in production only.

Because that, IMO, the best solution is a docker container that would deliver exactly the same libraries in any environment. Could be overkill in your case though.

If you are used to docker that is pretty straight forward:

RUN deps='gcc make'; 
    && apt-get update && apt-get install -y $deps --no-install-recommends 
    && pip install MySQL-python
    && apt-get purge -y --auto-remove $deps

The example do everything in one line to avoid the intermediate files to be bundled to the docker image.

I guess that would be also the straight approach if you ask about this in https://superuser.com/ instead. Here is a small tutorial if you are not used to docker:

Docker Explained: How To Containerize Python Web Applications

Community
  • 1
  • 1
olivecoder
  • 2,858
  • 23
  • 22
  • Yes, I could use a container (docker or other). But how to install psutil inside the container? – guettli Jun 02 '16 at 13:11
  • 1
    That isn't really a big deal. You could even use pip and uninstall the gcc later and lib headers later using a generic hardening script/tool. – olivecoder Jun 02 '16 at 14:55
  • somehow this looks like recursion to me. I could use the same pattern (install gcc, install psutil via pip, uninstall gcc) on the real host. But you are right. I could build the container in my environment and later deploy that container ... – guettli Jun 03 '16 at 04:50
  • 1
    Well... You can also use wheel of course. Using wheel and docker would have a similar effect. However I think that pip install and rpm/yum uninstall would be straight forward and easy to have in a dockerfile. – olivecoder Jun 03 '16 at 06:45