25

Coming from a Node.js + npm background, it is really nightmarish trying to understand all the things related to Python package management. After a few hours of research, I've stumbled upon all those keywords:

  • easy_install
  • virtualenv
  • pip
  • setuptools
  • distutils
  • pypi
  • wheel
  • egg
  • site-packages

Can someone help me decipher those terms and put them in historical context? For example, "distutils was the first package manager but it was superseded by X in Y because Z".

I absolutely love Python (the language) but package management seems like a real nightmare to learn for someone who has been using the amazing npm for the last few years.

mscdex
  • 104,356
  • 15
  • 192
  • 153
Icoin
  • 261
  • 2
  • 6
  • I did some Rails work a while back and really liked RubyGems:). For a number of reasons, I have to learn Python for my current project and was hoping someone could prove me wrong about package management being nightmarish as I really like the language. – Icoin Oct 30 '14 at 18:32
  • 4
    TL;DA version: Create a virtualenv using `virtualenv env`, enable it using `source env/bin/activate` in your shell, install packages (usually from pypi) using `pip install pakcagename` – ThiefMaster Oct 30 '14 at 18:36
  • in the case of [virtualenv](http://virtualenv.readthedocs.org/en/latest/) it is equivalent to the [RVM](http://rvm.io/) – neiesc Oct 30 '14 at 18:38
  • 1
    This post is NOT to say Ruby or Java, or whatever is better than Python. – DevLounge Oct 30 '14 at 18:49
  • @Apero no, just trying to make sense out of package management. As I said I really love the Python language. – Icoin Oct 30 '14 at 19:07
  • @Icoin this wasn't about your question but about the comments above – DevLounge Oct 30 '14 at 19:16
  • Eh I love python too but package management is... interesting. – ErlVolton Oct 30 '14 at 19:27

3 Answers3

21

Types of Packages
Egg vs Wheel vs Neither. What's meant by neither is that a python package can be installed from its "source" without being packaged as an egg or wheel.

Packaging Utilities
There are several libraries which provide utilities for packaging python applications, including distutils and setuptools. There is already an excellent post on this.

easy_install
Part of setuptools, allows building and installing python packages. Often discouraged in favor of Pip. Designed to make installation of packages easy, doing the chore of downloading and moving them to the correct place for you (hence the name).

Pip
A package manager for python packages, and a replacement for easy_install! See here for some reasons why people prefer it over easy_install. Can do neat things like install a package directly from a git repository or compile C extensions on the target machine. The latter is debatable as to whether or not it's desirable, but nonetheless it's a nice feature to have if you want it.

PyPI
The python package index, where easy_install and Pip search for available packages, by default. Basically a giant online repository of modules that are accepted by the community.

virtualenv
A way of hacking your environment variables to "isolate" an installation of python and it's related modules. Prefers Pip, because Ian Bicking wrote them both. Basically, you use pip to install virtualenv system wide, which then allows you to create python virtual environments, each with their own copy of python, pip, and assorted modules. This lets you have multiple versions of python or install a module just for testing, without mucking up your system-wide python install.

virtualenvwrapper
A really handy shell script that makes creating and tearing down virtual environments easier.

site-packages
One of the supported locations for installing python modules into. Lives someplace like /usr/lib/pythonX.X/site-packages. There are other supported locations, like dist-packages or user specific locations.

What does all this mean for you?
I'd recommend you don't pay any attention to easy_install and just use pip. Please also always use virtualenv. Usually, the only python modules you should install system-wide on your workstation are pip and virtualenv. I've completely ignored eggs and wheels, but if you plan to distribute packages professionally or host them on PyPI, you probably want to investigate those. Also, if you are creating python packages, you will need to learn to write a setup script, with setuptools. My recommendation is to never use distutils.

Some more Reading
A page on python.org about packaging which covers a lot of these topics
Python packaging is a nightmare
A great post that goes against the most common recommendations, including mine!

Community
  • 1
  • 1
ErlVolton
  • 6,714
  • 2
  • 15
  • 26
  • 1
    I also do not use eggs, but use egg-links a lot, inside virtualenv, whenever I install my package using python setup.py develop, it is really useful and allows you to just modify your python code, without having to reinstall it everytime. – DevLounge Oct 30 '14 at 19:22
  • Great answer. It seems that packages are built and uploaded to PyPi (setuptools / twine) using different tools than the tools that are used to manage/install packages (Pip / easy_install / virtualenv), unlike npm/gem. Is my understanding correct? – Icoin Oct 30 '14 at 19:36
  • Yes. It's also really confused by the fact that no one agrees on the best way to package, install, or manage python packages. Each of those three stages can be accomplished using many different tools, the most popular of which are not or were not part of the standard python library / supported by the PSF. See the last link in more reading for a great example on just how furious people get over this stuff. This is why I didn't include history in my answer, because it's just too much to handle! :) – ErlVolton Oct 30 '14 at 19:39
  • How do conda and brew fit in to all of this? – capybaralet Jul 31 '16 at 18:42
  • Conda is a binary package manager that was developed for use as a part of the Anaconda ecosystem and can be used to install and manage both system packages and python packages. If you are considering using Conda in your project, I suggest you first gain a strong knowledge of setuptools and pip, as that is a more common route. I imagine most of the people working with Conda are tied to Anaconda (https://docs.continuum.io/anaconda/) Brew is a thrid party package manager for OSX, and doesn't have much at all to do with python - @David – ErlVolton Aug 01 '16 at 19:39
3

There's some mixing in the options you are listing:

  • virtualenv - is used to create isolated environments

  • site-packages - standard location where python packages / libs reside

  • pypi - is a repository

  • easy_install - is found on the setuptools package

  • pip - was written to improve easy_install.

about python eggs

ZeroSoul13
  • 99
  • 2
  • 9
1

As of December 2018 the canonical way for distributing python software is following:

  • Use setuptools as the distribution library
  • Use wheel as the distribution format

$ python setup.py bdist_wheel

  • Once you have proper setup.py file, you can build a source tarball that can be distributed

    $ python setup.py sdist

The sdist command creates a tarball under the dist directory of the source tree. The tarball contains all the Python modules that are part of the source tree

  • The final step is to export your package somewhere users can install it via pip. That means publishing your project to PyPI.

make sure you have wheels installed in order to have bdist_wheel installed

pip install wheel
Vlad Bezden
  • 83,883
  • 25
  • 248
  • 179