2

I am trying to be able to run python2 and python3 simultaneously on a server that I do not have root access to and which does not have a recent python3 implementation. This works fine on my own server which I do have root access to, but I can't figure out what I am doing wrong.

For reference, the python2 implementation has been installed by administrators on the centos/rocks system in question, and they have both version 2.6.6 and 2.7.5. They also installed python 3.3.2, but there are some things in version 3.4 that I use, so I custom installed version 3.4.3 and put it in my PATH, LIBPATH, and PYTHONPATH. I use that python version exclusively for my own code, and I install libraries there with pip. I have created my own custom rocks module for the python3 version that sets the PATH, LIBPATH and LD_LIBRARY_PATH.

Everything works fine when I am just running python3 code. The problem comes when I try and simultaneously run python3 and python2 code in the same session.

I have written code in python3 that makes a system call to run a python2 program1. The problem is that in order to run my python3 code, I have my PYTHONPATH set to only contain python3 libraries. If python2 libraries are in that PATH, I get errors similar to this:

Failed to import the site module
Traceback (most recent call last):
  File "/opt/scipy/2.7/lib/python2.7/site-packages/site.py", line 73, in <module>
    __boot()
  File "/opt/scipy/2.7/lib/python2.7/site-packages/site.py", line 2, in __boot
    import sys, imp, os, os.path   
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/imp.py", line 22, in <module>
    from importlib import util
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/importlib/util.py", line 12, in <module>
    from contextlib import contextmanager
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/contextlib.py", line 4, in <module>
    from collections import deque
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/collections/__init__.py", line 17, in <module>
    from reprlib import recursive_repr as _recursive_repr
  File "/opt/biotools/qiime/lib/python2.7/site-packages/reprlib/__init__.py", line 8, in <module>
    raise ImportError('Cannot import module from python-future source folder')
ImportError: Cannot import module from python-future source folder

However, if I strip out all of the python2 stuff from the PYTHONPATH, then python2 scripts fail with errors like this:

File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/site.py", line 176
file=sys.stderr)
    ^
SyntaxError: invalid syntax

The only way I have found to avoid this is to include an explicit PYTHONPATH in the call to the python2 code. That does work, but it is hugely cumbersome and ugly.

I used to keep all of my python3 stuff in the PYTHON3PATH, and the python2 stuff in the PYTHONPATH, but now python3 appears to completely ignore the PYTHON3PATH and only use the PYTHONPATH.

Again, this problem doesn't occur on my own personal server, which runs arch linux and has python 3.4 and python 2.7 running side-by-side happily even though my PYTHONPATH contains directories for both.

I am sure I am doing something stupid here, but I don't know what it is.

1 Footnote to stave off silly comments: They python2 script is not written by me, the python3 code is a simple job management system that coordinates multi-threading none-parallel code on a remote machine. It is supposed to run any standalone program in parallel, but it is failing with python2 scripts. So I can't just switch one or the other to either version 2 or 3, and I can't merge the two by using 2to3 and importing.

Mike D
  • 727
  • 2
  • 10
  • 26
  • Why do you find that setting the PYTHONPATH to python2 for python2 execution is "hugely cumbersome and ugly"? It seems like a simple and correct solution to me. – antont Jun 30 '15 at 02:30
  • @antont The reason is that the script I wrote runs on a remote server with a complex python path, and that other people use this script. The server has a rocks module system, and so for us to use some bioinformatics modules with the python2 code that I am executing, I have to source 3 modules, and the python2 path ends up with 6 entries, and is 417 characters long. I would have to manually add that every time and only when running python2 code. That is a lot of work and would involve either hard coding or passing very long strings as arguments. I don't like that idea if I can avoid it. – Mike D Jun 30 '15 at 18:19
  • if you can have the path just as a configuration setting (or well hard coded in your code like you say), i don't see how it matters how many entries it has nor how many chars long it is. computers can deal easily even with megabytes you know :) so given that you can just put that conf somewhere and use it without needing to do anything manually it still seems like an ok solution to me. i certainly was not suggesting you'd manually type it in all the time.. sure there can be other options too, i don't know rocks etc. – antont Jul 01 '15 at 04:27
  • http://bugs.python.org/issue2375 – gavv Jul 17 '15 at 15:10
  • Maybe add PYTHON3PATH handling to python3 site.py? – gavv Jul 17 '15 at 15:11

1 Answers1

2

Calling python with the explicit path or version resolved the issue. For example:

$ python3.x
$ path_to/python
caot
  • 3,066
  • 35
  • 37
  • Yes this works well now. I have been using this method for several months without any issues. I also learned how to use the Rocks modules system on the cluster I was working with, that made the whole thing even simpler, but it only works on clusters based on Rocks. – Mike D Jan 11 '16 at 17:00
  • 1
    @Mike Environment Modules at http://modules.sourceforge.net/ also works well in linux and cluster. – caot Jan 11 '16 at 20:37