scipy ImportError on travis-ci

Question

I'm setting up Travis-CI for the first time. I install scipy in what I believe is the standard way:

language: python
python:
  - "2.7"
# command to install dependencies
before_install:
  - sudo apt-get -qq update
  - sudo apt-get -qq install python-numpy python-scipy python-opencv
  - sudo apt-get -qq install libhdf5-serial-dev hdf5-tools
install:
   - "pip install numexpr"
   - "pip install cython"
   - "pip install -r requirements.txt --use-mirrors"
# command to run tests
script: nosetests

Everything builds. But when the nosetests begin, I get

ImportError: No module named scipy.ndimage

Update: Here is a more direct demonstration of the problem.

$ sudo apt-get install python-numpy python-scipy python-opencv
$ python -c 'import scipy'
Traceback (most recent call last):
  File "<string>", line 1, in <module>

ImportError: No module named scipy

The command "python -c 'import scipy'" failed and exited with 1 during install.

I tried installing scipy using pip also. I tried installing gfortran first. Here is one example of a failed build. Any suggestions?

Another Update: Travis has since added official documentation on using conda with Travis. See ostrokach's answer.

Do you see any interesting messages when you remove the `-qq` option from `sudo apt-get install python-scipy`? If you open a terminal, can you `import scipy.ndimage`? — unutbu, Jun 20 '13 at 22:00
I removed ``-qq`` ([see latest build here](https://travis-ci.org/danielballan/mr/builds/8292380)) and I see no problematic messages. If I open a terminal on my computer, yes, I can ``import scipy.ndimage``. But maybe I misunderstood you: is there some way to open a terminal within Travis? — Dan Allan, Jun 20 '13 at 22:18
Sorry Dan, I don't know. Is it just scipy that raises an import error, or does numpy, cv, etc. raise `ImportErrors` too? — unutbu, Jun 21 '13 at 00:30
Good suggestion. Numpy is fine, but cv and cv2 also raise ImportErrors. I'm not sure what we've learned here, but it's not scipy's fault alone. — Dan Allan, Jun 21 '13 at 01:44
If you look inside the failed logs for the `pip` commands, you will see Travis CI is installing numexpr/cypthon/etc in a virtualenv environment. So my guess is that the `sudo apt-get install` commands are installing scipy and friends for the system's python, but the Travis CI environment is using the virtualenv's python for the nosetest, which does not have access to system's packages. I know you already tried installing scipy using pip, but perhaps try it again using [this](http://blog.adamdklein.com/?p=416) as a guide. — unutbu, Jun 21 '13 at 10:58
relevant Travis issue: https://github.com/travis-ci/travis-ci/issues/2650 — 0 _, Jan 03 '16 at 14:40

Dan Allan · Accepted Answer · 2014-09-19T16:02:26.503

I found two ways around this difficulty:

As @unutbu suggested, build your own virtual environment and install everything using pip inside that environment. I got the build to pass, but installing scipy from source this way is very slow.
Following the approach used by the pandas project in this .travis.yml file and the shell scripts that it calls, force travis to use system-wide site-packages, and install numpy and scipy using apt-get. This is much faster. The key lines are
```
virtualenv:
    system_site_packages: true
```
in travis.yml before the before_install group, followed by these shell commands
```
SITE_PKG_DIR=$VIRTUAL_ENV/lib/python$TRAVIS_PYTHON_VERSION/site-packages
rm -f $VIRTUAL_ENV/lib/python$TRAVIS_PYTHON_VERSION/no-global-site-packages.txt  
```
and then finally
```
apt-get install python-numpy
apt-get install python-scipy
```
which will be found when nosetests tries to import them.

Update

I now prefer a conda-based build, which is faster than either of the strategies above. Here is one example on a project I maintain.

If `pip install scipy` timeouts on Travis, then the timeout can be increased as described here: http://stackoverflow.com/questions/28746046/how-can-i-install-something-on-travis-ci-without-a-timeout — 0 _, Jan 03 '16 at 14:32
Using `sudo` also means that the newer, container-based infrastructure of travis is not used. Besides, the system-provided `numpy` and `scipy` are quite outdated. — 0 _, Apr 11 '16 at 08:38

score 5 · Answer 2 · answered Jul 21 '15 at 02:18

This is covered in the official conda documentation: Using conda with Travis CI.

The .travis.yml file

The following shows how to modify the .travis.yml file to use Miniconda for a project that supports Python 2.6, 2.7, 3.3, and 3.4.

NOTE: Please see the Travis CI website for information about the basic configuration for Travis.

language: python
python:
  # We don't actually use the Travis Python, but this keeps it organized.
  - "2.6"
  - "2.7"
  - "3.3"
  - "3.4"
install:
  - sudo apt-get update
  # We do this conditionally because it saves us some downloading if the
  # version is the same.
  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
      wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
    else
      wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
    fi
  - bash miniconda.sh -b -p $HOME/miniconda
  - export PATH="$HOME/miniconda/bin:$PATH"
  - hash -r
  - conda config --set always_yes yes --set changeps1 no
  - conda update -q conda
  # Useful for debugging any issues with conda
  - conda info -a

  # Replace dep1 dep2 ... with your dependencies
  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION dep1 dep2 ...
  - source activate test-environment
  - python setup.py install

script:
  # Your test script goes here

Works great, and very fast too. – Ami Tavory Jul 12 '17 at 05:31 — Ami Tavory, Jul 12 '17 at 05:31

score 4 · Answer 3 · answered Sep 24 '13 at 16:17

I found this approach to work:

http://danielnouri.org/notes/2012/11/23/use-apt-get-to-install-python-dependencies-for-travis-ci/

Add these lines to your Travis configuration to use a virtualenv with --system-site-packages:

virtualenv:
  system_site_packages: true

You can thus install Python packages via apt-get in the before_install section, and use them in your virtualenv:

before_install:
 - sudo apt-get install -qq python-numpy python-scipy

A real-world use of this approach can be found in nolearn.

This doesn't work anymore. Check this one instead: https://github.com/strawlab/pymvg/commit/42b66c69348145c5a99c058e769c674af0749713 — Hakim, Sep 23 '15 at 22:26

score 1 · Answer 4 · answered May 11 '15 at 18:48

As Dan Allan pointed out in his update, he now prefers a conda-based build. Here is a gist courtesy Dan Blanchard giving a full .travis.yml file example that will pre-install scipy on the test machine:

language: python
python:
  - 2.7
  - 3.3
notifications:
  email: false

# Setup anaconda
before_install:
  - wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
  - chmod +x miniconda.sh
  - ./miniconda.sh -b
  - export PATH=/home/travis/miniconda/bin:$PATH
  - conda update --yes conda
  # The next couple lines fix a crash with multiprocessing on Travis and are not specific to using Miniconda
  - sudo rm -rf /dev/shm
  - sudo ln -s /run/shm /dev/shm
# Install packages
install:
  - conda install --yes python=$TRAVIS_PYTHON_VERSION atlas numpy scipy matplotlib nose dateutil pandas statsmodels
  # Coverage packages are on my binstar channel
  - conda install --yes -c dan_blanchard python-coveralls nose-cov
  - python setup.py install

# Run test
script:
  - nosetests --with-cov --cov YOUR_PACKAGE_NAME_HERE --cov-config .coveragerc --logging-level=INFO

# Calculate coverage
after_success:
  - coveralls --config_file .coveragerc

scipy ImportError on travis-ci

4 Answers4

The `.travis.yml` file

scipy ImportError on travis-ci

4 Answers4

The .travis.yml file

The `.travis.yml` file