2

In a Linux compute cluster environment that we have we would previously install different versions of python that we needed under /opt on the cluster headnode, and configure /etc/exports to NFS export /opt/pythonX.Y to the compute nodes for mounting from the headnode. However, we recently moved to a more resilient storage location on a SAN that is mounted on the whole cluster. We rsynced, for example, /opt/python2.7 and /opt/python3.3 from the headnode to the new path on the mounted SAN but the problem is that many of the *.py files under their bin/ subdirectory still have a shebang pointing to the old /opt path, thus keeping that as a dependency.

I thought that it would be easy to just freshly install python on the SAN instead by dumping the current packages that are in the /opt python installs with 'pip freeze > packagelist.txt' and then reinstalling with 'pip -r packagelist.txt', but what that did was install all of the eggs (expected) but almost none of the corresponding bin scripts (unexpected). I quickly discovered this when users started complaining that their bash scripts which use many of those *.py bins/scripts broke when their .bashrc was pointing to the new SAN path to run python, primarily because all the *.py scripts (which are under the /opt install path) were not under the SAN's pythonX.Y/bin/, although the corresponding eggs were all installed to the site-packages directory.

I have been trying to understand the relationship between the eggs and scripts as regards an install and the directory (output) location and I tracked down that the setup.py must have a 'scripts=' parameter specified in the setup() function as mentioned in this SE question: setup.py and adding file to /bin/.

I am a non-developer and do not even code in python, but can someone please help me understand why a pip "reinstall" using freeze/-r (--requirement) would install the eggs to site-packages/ but not the scripts to bin/? If they didn't come from the same package or install then where did they come from? If you need more specifics please let me know.

Community
  • 1
  • 1
SeligkeitIstInGott
  • 223
  • 2
  • 7

1 Answers1

0

It could be that you've rsynced over a distribution configuration file for python. Specifically pydistutils.cfg:

https://docs.python.org/2/install/#location-and-names-of-config-files

To clarify, scripts refer to python programs typically put on the PATH while "eggs" or otherwise refer to python modules and libraries used in python programs.

In pydistutils.cfg there should be a setting for "install_scripts" or "script_dir". I'd wager it's still pointing at your old python script directory if this is the case.

EricR
  • 579
  • 5
  • 10
  • I could be looking in the wrong area but there are no *.cfg files in /python2.7/lib/python2.7/distutils/ even though there is a config.py. I would like it much better if I could just start from scratch with a fresh install though. So to your knowledge should pip install both the python egg libraries/modules and the script executables to go on my PATH? If so are there any caveats to using pip freeze/-r or should I just get an xargs list of all the current packages installed under the /opt path (a few hundred packages actually) and just do a normal pip install? Thanks. – SeligkeitIstInGott Apr 14 '15 at 15:41
  • And another question about pip, will pip freeze dump all installed python packages even if they had been installed with easy_install or even directly with 'python setup.py'? My pip freeze list did look a little short, so I'm wondering if it is not picking everything up. – SeligkeitIstInGott Apr 14 '15 at 15:45
  • The configuration file may or may not exist. – EricR Apr 14 '15 at 17:37
  • Your method of pip freeze then pip install -r requirementlist.txt is sound. The only thing that raises an eyebrow is the fact that the scripts are installing to the older script path (from what I understand). pip freeze should dump all packages it can find on it's python package path. These packages are both found through $PYTHONPATH and on the directory which python is being run from. Since pip comes default with newer installations are you sure you're running pip for your older installation of python? – EricR Apr 14 '15 at 17:47
  • "The only thing that raises an eyebrow is the fact that the scripts are installing to the older script path (from what I understand). pip freeze should dump all packages it can find on it's python package path." Sorry if I was not clear. It is not installing the scripts to the old location, it is going where they are supposed to. The problem is that it is installing substantially less scripts than are in the old location, making me wonder why some packages seem to only be installing the eggs but not the scripts. Perhaps newer version of the packages have different scripts? – SeligkeitIstInGott Apr 14 '15 at 20:07
  • It could be the case that the scripts do not actually exist anymore. Are you 100% sure that it's the same list of packages from your previous install? Is there a package that you know has a script in your previous install that doesn't exist in your newer one? Take a look at: https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation . You can look through the setup.py file to see if you can find a counterexample where a script from a package is not being installed correctly. If that's the case then it's probably not a python-specific problem. – EricR Apr 15 '15 at 14:57
  • I'm doing a deep dive into specific scripts in the old /opt location that do not appear in the new install location to see what packages they all belong to. There are 125 scripts out of 333 total missing. It has been a while since I tried a fresh install (this was sitting on my back burner long before I posted the question) so I'll try again and see if I can get more specific details on the issue. Thanks for your help thus far. – SeligkeitIstInGott Apr 16 '15 at 20:41
  • I discovered the main package that was missing: bx-python. It accounted for about 100 of the 125 missing scripts, and it is perfectly accessible through pip (unlike the other package with the remaining 25 scripts which has to be downloaded and manually installed from source from its setup.py). The only conclusion that I can reach is that easy_install was used to install bx-python and that a pip freeze will only dump packages that it had installed to begin with - which is extremely inconvenient for reinstalls/migrations. Oh well. – SeligkeitIstInGott May 06 '15 at 01:44