4

(Advanced apologies for the lack or proper links; the system won't allow me to add more than two.)

Unfortunately, I've learnt the hard way that you shouldn't mess with the default Python installations in Mac OS X (specifically, 10.6.8).

After using the python.org installers for 2.6.6 (http://www.python.org/getit/releases/2.6.6/) and 2.5.4 (http://www.python.org/getit/releases/2.5.4/), I have Python versions which are more mature than those provided by Apple (which is great for development), but have broken core system functionality (which is bad for just about everything else.) The most visible breaks so far have been when trying to run namebench (https://code.google.com/p/namebench/), Blink (http://icanblink.com/) and Mercurial (https://www.mercurial-scm.org/). From what I can gather, it's down to the paths.

In a default installation, the paths should resemble something those outlined in this question. Instead, mine look like this:

$ /usr/bin/python2.6 -V
Python 2.6.6

$ which python
/usr/bin/python
$ python
Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for i in sys.path:
...     print i
... 
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pip-1.0.2-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages

(It's a similar story for 2.5 but, for simplicity, I'll stick with 2.6.)

The issue seems to be the non-inclusion of:

  • /Library/Python/2.6/site-packages/*.egg
  • /Library/Python/2.6/site-packages
  • /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
  • /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python

This explains why Blink and namebench can't find PyObjC (Is PyObjC pre-installed on OSX SL?) and Mercurial can't find its modules (in /Library/Python/2.6)

Issue #4865 in the Python bug tracker partially addresses this particular problem, which has since been fixed for versions 2.7 and greater. But because it's "a feature request and not a bugfix", has not been back-ported to 2.5 and 2.6. Even then, looking at the committed fix (http://hg.python.org/lookup/r70778), I'm not sure it addresses the lack of references to the "Extras" directories.

I understand that I can manually add paths to Python by manually altering site.py. I could also make use of the PYTHONPATH environment variable. I'd rather not cause any further damage by altering site.py, and changes PYTHONPATH would only be valid for scripts/applications run from the shell using my user account.

Can I get these more recent versions of Python to reference the same paths as the default framework installations? If so, what is the best way to go about it? If not, is there an accepted method of rolling back to the system defaults?

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
coffeebucket
  • 43
  • 4
  • coffeebucket: Welcome to SO! While extra information is nice, some links in your post are unnecessary. You don't need to give every download/project page link for everything, if further clarification is needed someone can ask for it in the comments later on anyway. – orlp Aug 20 '11 at 07:00
  • Thanks nightcracker. I may have got a little carried away with the extra information. Duly noted for future posts. – coffeebucket Aug 20 '11 at 07:05

1 Answers1

3

You misunderstand how Python installations work on OS X. Each Python instance has its own site-packages directory. The standard location for framework installers is within the framework at ./lib/pythonx.y/site-packages. So for the python.org installers which install into /Library/Frameworks/Python.framework, you will find its 2.6 site-packages here:

/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/

Apple makes some modifications to the versions of Python that it ships with OS X releases. From OS X 10.5 on, the system Pythons are installed at:

/System/Library/Frameworks/Python.framework

and Apple chooses to include some extra 3rd-party packages in the non-standard ./Extras directory within each version. It also uses a non-standard location for each version's site-packages directory. They are created in /Library/Python/, presumably so that user-installed packages do not modify anything in /System/Library. So for the Apple-supplied Python 2.6, its site-packages directory is:

/Library/Python/2.6/site-packages

and can be thought of as having been extended by the packages in ./Extras.

Each Python instance has a separate site-packages directory. It is not intended and often not possible to share packages across site-packages of different instances even of the same minor version of Python, i.e. 2.6. The most obvious problem is that there are often differences in the C compiler version, the OS X ABI (MACOSX_DEPLOYMENT_TARGET), the SDK version, and/or the CPU architectures used to build the Python interpreters and which are subsequently used by Distutils to build C extension modules included in 3rd-party packages.

In Mac OS X 10.6, the Apple-supplied Python is built using gcc-4.2 and targeted just for OSX 10.6 and includes 3 CPU architectures (i386, x86_64, and ppc). The python.org installers for Python 2.6 are built to run on older systems as well, so have a target of 10.3 and later, use gcc-4.0, and are 32-bit only (i386 and ppc). So, in general, you cannot run C extension modules built for the one Python with the other.

This means that, in general, you need to install separate copies of 3rd-party packages you need with and for each Python, if they are not already included with that Python. This includes basic items like easy_install (from setuptools or Distribute). The system Pythons in 10.5+ include easy_install version in /usr/bin for them. If you install a python.org Python, you'll need to install a separate version for it; by default, the easy_install command will be installed in that Python's ./bin directory in its framework; that is the Distutils default location. That's why it is recommended that you add this directory to your shell PATH (and the python.org installer for Python 2 do that automatically by default).

The change introduced by Issue4865 is not really a good solution and may fail with C extension modules. I would not depend on it remaining in Python in future versions.

Also installing a python.org Python in no way breaks anything in the system Python because they are completely independent installations using different file system locations. The only thing that may change is which instance of Python is invoked when you type a particular name. That is controlled primarily by the search order of your shell PATH environment variable. As noted, the python.org installer by default changes that order but the system Python is still readily available by using its absolute path /usr/bin/python2.6. Or you can revert the changes to shell profile, for instance, .bash_profile.

$ echo $PATH
/Library/Frameworks/Python.framework/Versions/2.6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/Library/Frameworks/Python.framework/Versions/2.6/bin/python
/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
$ python -V
Python 2.6.6
$ python2.6 -V
Python 2.6.6
$ /usr/bin/python -V
Python 2.6.1
$ /usr/bin/python2.6 -V
Python 2.6.1
#
# remove python.org Python 2.6 from PATH
#
$ export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/usr/bin/python
/usr/bin/python2.6
$ python -V
Python 2.6.1
$ python2.6 -V
Python 2.6.1
Ned Deily
  • 83,389
  • 16
  • 128
  • 151
  • Thanks for your detailed clarification. To save recreating Apple's Python "environment", I guess it would be easiest to remove the python.org versions. I've just I come across your instructions on how to manually remove python.org installations ([Issue7107](http://bugs.python.org/issue7107)), but I have a few reservations. In particular, on my system `/Library/Frameworks/Python.framework` is a symlink to `/System/Library/Frameworks/Python.framework/`. Am I missing something obvious? Let me know if I should create a separate question for this issue. – coffeebucket Aug 20 '11 at 08:38
  • `/Library/Framework/Python.framework` should not be a symlink to `/System/Library/Frameworks/`. I'm not sure what you might have installed that did that. There should be no danger in removing it. But if that is the case, then you do not the python.org version installed any more, since it has apparently been overwritten by this symlink (and the demo at the end of my answer above couldn't work for you). – Ned Deily Aug 20 '11 at 09:11
  • Looking at another non-development 10.6.8 machine I have, the link definitely does not exist by default. I'm not exactly sure how it happened. From the symlink's timestamp, it appears to have been created around the same time I ran the 2.5.4 installer. Then the 2.6.6 installer has installed over the top of the 2.6.1 framework installation. Either way, this is bad news. Amazingly, Time Machine has copies of 2.5 and 2.6 before I performed the python.org installations. If you have any other suggestions as to how I could recover from this, I'd be very happy to hear them. – coffeebucket Aug 20 '11 at 09:45
  • If you have Time Machine backups of the original /System/Library/Frameworks/Python.framework (with the Apple 2.6.1 and 2.5.4), I would just delete the faulty symlink, then `mv` the overwritten `/System/Library/Frameworks/Python.framework` to `/Library/Frameworks/Python.framework/` and then restore `/System/Library/Frameworks/Python.framework` from the Time Machine backup. You should then be in pretty good shape, I'd think. – Ned Deily Aug 20 '11 at 09:58
  • P.S. I think I've found one possible reason why that symlink was there: someone followed the questionable advice from this page in the Python Wiki http://wiki.python.org/moin/MacPython/Leopard . I've added a warning for the time being. That page is out-of-date anyway. – Ned Deily Aug 20 '11 at 20:52
  • Thanks again for the additional information. I'm certain I wouldn't have created the link of my own volition, and it was most likely the 2.5 installer that did it. I've extracted a few different snapshots of `/System/Library/Frameworks/Python.framework` from Time Machine and am narrowing down a point before the damage was done. Aside from `python`, `pythonw` and possibly `python2.6`, are there any other files I need to restore to `/usr/bin`? – coffeebucket Aug 21 '11 at 13:42
  • Possibly `2to3`, `idle`, `pydoc`, and `python-config`. Like `python` and `pythonw`, they are all wrapper programs for their versioned symlink files (`pydoc2.6`, etc). – Ned Deily Aug 21 '11 at 17:44
  • After restoring a "good" copy of `Python.framework`, the applications listed in my original question now work and everything seems to be back to normal. The references to `python`, `pythonw`, `2to3`, `idle` and `pydoc` all seem to have resolved themselves without me having to restore older versions. I'm keeping a restored Time Machine snapshot on hand should I need it. (`python2.6` references a MacPorts install, but that's a different matter entirely.) Now to configure `virualenv` and restore the environment I had set up before. Thanks again for all your help; it's very much appreciated. – coffeebucket Aug 24 '11 at 11:14