32

I am using numpy / scipy / pynest to do some research computing on Mac OS X. For performance, we rent a 400-node cluster (with Linux) from our university so that the tasks could be done parallel. The problem is that we are NOT allowed to install any extra packages on the cluster (no sudo or any installation tool), they only provide the raw python itself.

How can I run my scripts on the cluster then? Is there any way to integrate the modules (numpy and scipy also have some compiled binaries I think) so that it could be interpreted and executed without installing packages?

Timothy
  • 4,467
  • 5
  • 28
  • 51
  • I'm not really clear on exactly what you're asking--most Python modules are just Python code that you call `import` on. If that's the issue, and you can't use `pip` or `easy_install`, you can just download the packages and stick them right in your Python `site-packages`, and then just `import` them. – jdotjdot Jan 06 '13 at 06:39
  • @jdotjdot there're not only python code, but also compiled binary in the modules. – Timothy Jan 06 '13 at 06:42
  • 3
    For `numpy` and `scipy` that is true. But what do you mean by "can't install packages"? What exactly is it that you're not permitted to install/put on the cluster? – jdotjdot Jan 06 '13 at 06:43
  • @jdotjdot we're not permitted to `sudo`, and there's not any installation tool either. – Timothy Jan 06 '13 at 06:51
  • thanks, that's what I was looking for. Looks like **David** provided a good answer already. – jdotjdot Jan 06 '13 at 07:15
  • @jdotjdot Thanks for making my question clear. – Timothy Jan 06 '13 at 13:32
  • Does the server have `pip` or `easyinstall`? – Colonel Panic Jan 06 '13 at 14:46
  • @ColonelPanic No, no any installation tool. – Timothy Jan 06 '13 at 15:52
  • scipy might not be that easy to install, if the required dependencies (outside of python) are not installed – Fábio Dias Mar 10 '16 at 20:58

4 Answers4

44

You don't need root privileges to install packages in your home directory. You can do that with a command such as

pip install --user numpy

or from source

python setup.py install --user

See https://stackoverflow.com/a/7143496/284795

The first alternative is much more convenient, so if the server doesn't have pip or easy_install, you should politely ask the admins to add it, explaining the benefit to them (they won't be bothered anymore by requests for individual packages).

Community
  • 1
  • 1
Colonel Panic
  • 132,665
  • 89
  • 401
  • 465
  • 9
    **politely ask the admins to add it**, why didn't I think of that! OMG, we geeks suck... – Timothy Jan 06 '13 at 16:02
  • 3
    @Skyler: asking the admins is *definitely* the best option! Based on your question I thought they might be opposed to it. (I do still recommend using virtualenvs once pip is installed, which incidentally might also serve as a solution to [this other question of yours](http://stackoverflow.com/questions/14117945/too-many-different-python-versions-on-my-system-and-causing-problems)). – David Robinson Jan 06 '13 at 17:04
  • 1
    This should have been `python setup.py install --user`. See https://pythonhosted.org/setuptools/easy_install.html#id37 – Piotr Dobrogost Oct 22 '14 at 22:04
12

You could create a virtual environment through the virtualenv package.

This creates a folder (say venv) with a new copy of the Python executable and a new site-packages directory, into which you can "install" any number of packages without needing any kind of administrative access at all. Thus, activating the environment through source venv/bin/activate will give Python an environment that's equivalent to having those packages installed.

I know this works for SGE clusters, although how the virtual environment is activated might depend on your cluster's configuration.

You can try installing virtualenv on your cluster within your own site-packages directory using the following steps:

  1. Download virtualenv from here, put it on your cluster

  2. Install it using setup.py to a specific, local directory to serve as your own site-packages:

    python setup.py build
python setup.py install --install-base /path/to/local-site-packages

  3. Add that directory to your PYTHONPATH:

    export PYTHONPATH="/path/to/local-site-packages:${PYTHONPATH}"

  4. Create a virtualenv:

    virtualenv venv
David Robinson
  • 77,383
  • 16
  • 167
  • 187
  • Thank you! How to install `virtualenv` without `sudo` or `pip` or `easy_install`? – Timothy Jan 06 '13 at 06:55
  • @Skyler: You can install `virtualenv` on a system where you do have control, create the virtualenv there, and then transfer the files to the cluster (unless I'm mistaken, and I could be, you don't even need `virtualenv` to activate the environment, only to create it- the environments is activated using bash scripts). *However*, that works only if the system you are creating the virtualenv on is of the same OS and architecture as the cluster- you might need to do a lot of customizing. – David Robinson Jan 06 '13 at 07:01
  • @Skyler: On second thought, it might be easier to install `virtualenv` in a local site-packages directory (basically bootstrapping your way up based on what you can do). Try the approach in my edit (I haven't tested it at all so let me know what problems you run into on the way). – David Robinson Jan 06 '13 at 07:12
  • Hi David, the step 2 gives `error: install-base or install-platbase supplied, but installation scheme is incomplete`. Did I miss something? – Timothy Jan 06 '13 at 14:30
  • You can also install the package virtualenvwrapper, it will install some new commands and you can create your virtualenvs with mkvirtualenv *envname*, and it will automatically activate it for you. http://www.doughellmann.com/projects/virtualenvwrapper/ – iferminm Jan 06 '13 at 15:07
  • 2
    @Skyler: Sorry- try replacing `--install-base` with `--home` – David Robinson Jan 06 '13 at 17:02
  • @DavidRobinson BTW, some of the dependencies have been already provided with the cluster. I want to create a virtualenv by cloning the current local environment, so that I don't have to install them again. Is it possible? I posted another question for this: http://stackoverflow.com/questions/14185288/how-to-create-a-virtualenv-by-cloning-the-current-local-environment – Timothy Jan 06 '13 at 18:35
  • @Skyler: unless you use the `--no-site-packages` option, the virtualenv already "inherits" all of the packages and dependencies installed on the full system by default. (Try it on your computer). – David Robinson Jan 07 '13 at 01:52
  • @DavidRobinson Does it inherit in a "copy" way? What if the outside environment changes in the future? P.S.: the `--home` works. and before step 4, the local path should also be added to `PATH` so that the `virtualenv` command could run. – Timothy Jan 07 '13 at 06:51
  • @Skyler: I'm not 100% positive, but I think it does copy them. – David Robinson Jan 07 '13 at 06:57
  • For me this works with: "python setup.py install --prefix=" This installs virtualenv's scripts to /bin and the package goes to /lib/python2.7/site-packages For some reason, using --home instead of --prefix puts the package into /lib/python/site-packages (without the python version number!) – Amit Moscovich May 01 '14 at 16:52
1

You can import a module from an arbitrary path by calling:

sys.path.append()

Brian Lyttle
  • 14,558
  • 15
  • 68
  • 104
john graham
  • 11
  • 1
1

The Python Distribution Anaconda solves many of the issues discussed in this questions. Anaconda does not require Admin or root access and is able to install to your home directory. Anaconda comes with many of the packages in question (scipy, numpy, sklearn, etc...) as well as the conda installer to install additional packages should additional ones be necessary.

It can be downloaded from https://www.continuum.io/downloads

Brian Cain
  • 946
  • 1
  • 7
  • 20