72

During the past years, I have installed many Python libraries with various Python versions. To make them ready to work immediately, I installed them blindly without control. Currently they're causing problems when I tried to install pynest which invokes numpy, scipy and matplotlib. After struggling, I am going to clean and reinstall Python and the libraries.

After investigation, I found Python 2.5/2.6/2.7/3.2 on my system, and each of them has some copies or other things at: (my OS == Mac OS X 10.7.5 Lion)

  • /Library/Frameworks/
  • /opt/local/Library/Frameworks/
  • /opt/local/bin/
  • /Applications/
  • /usr/local/bin/
  • /usr/bin/
  • /System/Library/Frameworks/

I know I'm crazy to have these. Now I have removed all these except the things in /System/Libarary/Frameworks (I never remove any thing from /System/Library/). After the clean work, which python now gives /usr/bin/python which links to /System/Library/Frameworks.

Now, is it a clear environment for me to reinstall python? How to double check that there's no other versions existing? How should I reinstall them to guarantee that they and their libraries won't be everywhere and have many copies again?

I want to install a clean Python 2.7 onto a proper location, and make my system know exactly where it is and never install any libraries somewhere else. Please give me some advice that how to manage it like in a professional way.

For your information, here is my current $PATH, I think it should be modified:

/opt/local/bin:/opt/local/sbin:/opt/nest/lib/python2.7/site-packages:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/django/bin:/usr/X11/bin:/opt/local/bin:/opt/local/sbin:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/3.2/bin

Please let me know If you need more information. Thank you!

UPDATE:

I'm rethinking profoudly why it becomes so crazy. I believe it's because I installed things via:

  • easy_install / macports / homebrew / fink / pip sometimes;
  • .dmg sometimes;
  • .pkg sometimes;
  • compile source code sometimes;

and they made things at different locations. I wonder what's the mechanism behind these ways? How do they choose target location? How to prevent them from messing things up?

Timothy
  • 4,467
  • 5
  • 28
  • 51
  • 1
    I suppose a full OS re-install is out of the question? It sounds like you need to look into [virtualenv](http://www.virtualenv.org/). – Nick Jan 02 '13 at 06:51
  • I don't know about Macs, but isn't there some kind of a package manager to uninstall it just like you installed it? Or did you just unzip it and edited the PATH and that's it? – jadkik94 Jan 02 '13 at 08:08
  • 1
    @jadkik94 yes there is. Sometimes I use `easy_install`/`macport`/`homebrew`/`fink`/`pip`, sometimes I use `.dmg`, sometimes I use `.pkg` and sometimes I compile source code to install different things. That's why it becomes so crazy. – Timothy Jan 02 '13 at 08:22

4 Answers4

37

Why did it get messed up?

There're a couples of different way to install Python, as the update of OP says, and they locate files in different locations. For example, macports puts things into /opt/local/, while homebrew puts things into /usr/local/. Also, Mac OS X brings a few python versions with itself. So, if you install python many times via different ways, you will get many python versions existing independently on your system.

What problem does it cause?

I don't know exactly. I guess the problem is that if you have many versions of python, then which one to use and where to find packages will be determined by the path order in your system PATH and the PYTHONPATH respectively. So you may lose control of where to install python modules. Consider that if you run sudo python setup.py install to install a module (it finds python by the root's PATH) and then try to import the module by python -c "import it" (this time it finds python by your PATH), maybe something will go wrong. This is my guess, I didn't validate it. But in my own case, something did go wrong.

How to avoid this?

I think the principle would be that be aware of that different ways and tools install things independently to different locations, so use them mindfully.

  • Unless you intend to, don't install the same thing twice via different ways. (If you intend to do it for python, you might want to check out virtualenv)
  • Keep an eye on the path order in your PATH and consider if it's correct.
  • When installing modules, be clear which python (or pip) is running and where the module is installed.

So, how did I solve my own case?

Since it had been messing up already and seemed to be very hard to cure, so finally I solved this question by a full OS re-installation, and started to follow the DOs-and-DONTs above. For the installation of the scientific environment with python (numpy/scipy/matplotlib, which had shown problems to make me ask this question), I found this tutorial was extremely helpful. So, problem solved finally.

RichVel
  • 7,030
  • 6
  • 32
  • 48
Timothy
  • 4,467
  • 5
  • 28
  • 51
  • thanks for the insights and recommendations. I seem to be trapped in a similar situation, but it's not getting unmanageable... yet. I've found a useful post: https://leemendelowitz.github.io/blog/how-does-python-find-packages.html that helps finding where a particular version of python looks for imports and basically manually made them search in the right place for the right thing via symbolic links. – alisa Nov 07 '16 at 22:04
  • 1
    Also, don't blindly follow guides on the internet in regards to getting something working. So stick with homebrew or macports depending on your preference.. don't use both. Don't use dmg or pkg to install certain packages if you have homebrew fx. stick with homebrew no matter what.. If you have multiple things at the moment, try looking into just uninstalling them. – Alex R Aug 09 '17 at 09:07
15

Here is what was confusing me and how I solved it.

$ which python
/usr/bin/python 

$ which python3
/usr/local/bin/python3

$ ls /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory

So notice I didn't have a HomeBrew installation of python2.7, but did have the python3 installation. The version under /usr/bin/python is using the system default. You can tell based on the module search path:

$ /usr/bin/python
Python 2.7.10 (default, Feb  7 2017, 00:08:15) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
`enter code here`Type "help", "copyright", "credits" or "license" for 
more information.
>>> import sys
>>> sys.path
['', '/Library/Python/2.7/...

Notice the '/Library/Python'... that's Mac OS's version of python. But I want to stay strictly on a user installed version (i.e. HomeBrew).

So here's what I did to fix this:

$ brew install python
...
Warning: python 2.7.13 is already installed, it's just not linked.
You can use `brew link python` to link this version.

$ brew link --overwrite python

$ which python
/usr/local/bin/python

$ python
Python 2.7.10 (default, Feb  7 2017, 00:08:15) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python/2.7.13...

Its no longer /Library/.. but /usr/local.

Now its finding all of my pip installed modules! Problem solved!

UPDATE:

After updating brew to version 1.5.4, it seems the symbolic links were removed. And now you have to add this to your path:

export PATH="/usr/local/opt/python/libexec/bin:$PATH"

Read the Caveats section in 'brew info python':

==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.bash_profile:
  export PATH="/usr/local/opt/python/libexec/bin:$PATH"

Pip and setuptools have been installed. To update them
  pip2 install --upgrade pip setuptools

You can install Python packages with
  pip2 install <package>

They will install into the site-package directory
  /usr/local/lib/python2.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python.html
jersey bean
  • 3,321
  • 4
  • 28
  • 43
8

In order to install a python distributions into specific folder, you can use the --prefix scheme during python installation. Using the prefix scheme, you can for example install Python 2.7 into the folder /opt/py27. Now, in order to use the new installed Python distribution you have to: cleanup you PATH and LD_LIBRARY_PATH:

  • Remove all 'old' Python paths and
  • configure (according to my example) the environment variables like this:
    • PATH: Add /opt/py27/bin
    • LD_LIBRARY_PATH: Add /opt/py27/lib

That's it.

(In case you need multiple environments of Python installed at the same time, I'd suggest to have a look at virtualenv)

gecco
  • 17,969
  • 11
  • 51
  • 68
  • I did what you said but the problem remains. It throws errors when I `import nest` and I believe it's caused by the multiples copies of libraries. How could I completely remove all the libraries(and also easy_install etc.) and reinstall? – Timothy Jan 02 '13 at 09:14
  • This sounds promising, but the page to which you link only discusses installing Python *modules* rather than the base system. How would you install the *whole* of Python2.7 into `/opt/py27` as you mention? It seems to me that the confusion comes principally because each and every Python version/flavour wants to install its own *binaries* (`python`, `pip`, `easy_install`) with the same names and in the same places as other versions/flavours. – jez Dec 17 '15 at 18:49
1

tl;dr
brew install python

Symptoms I had similar issues with python programs not finding dependencies.
My python3 version was a broken symlink.
My pip was pointing to a python 3.8
And my pip3 was pointing to 3.9
python -V was outputting some python 2.7 version
python3 -V was outputting some python3.8 version

Solution
I ran brew install python and it fixed all my problems.

13013SwagR
  • 433
  • 4
  • 13