1

Pip doesn't seem to build dependencies from source on my Ubuntu server, while it always does that on my OS X machine. For example, when I try to install package qiime in a conda or virtualenv (I tried both) environment it takes seconds to install a hell lot of things that take loads of time to compile on my Mac.

(qiime)user@server:~$ pip install qiime
Collecting qiime
Collecting qiime-default-reference<0.2.0,>=0.1.2 (from qiime)
Collecting burrito<1.0.0,>=0.9.1 (from qiime)
Collecting pandas>=0.13.1 (from qiime)
Collecting natsort<4.0.0 (from qiime)
  Using cached natsort-3.5.6-py2.py3-none-any.whl
Collecting matplotlib!=1.4.2,>=1.1.0 (from qiime)
Collecting numpy>=1.9.0 (from qiime)
Collecting gdata (from qiime)
Collecting scikit-bio<0.3.0,>=0.2.3 (from qiime)
Collecting pynast==1.2.2 (from qiime)
Collecting biom-format<2.2.0,>=2.1.4 (from qiime)
Collecting burrito-fillings<0.2.0,>=0.1.1 (from qiime)
Collecting qcli<0.2.0,>=0.1.1 (from qiime)
Collecting scipy>=0.14.0 (from qiime)
Collecting cogent==1.5.3 (from qiime)
Collecting emperor<1.0.0,>=0.9.51 (from qiime)
Collecting six (from qiime-default-reference<0.2.0,>=0.1.2->qiime)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting future (from burrito<1.0.0,>=0.9.1->qiime)
Collecting pytz>=2011k (from pandas>=0.13.1->qiime)
  Using cached pytz-2015.7-py2.py3-none-any.whl
Collecting python-dateutil (from pandas>=0.13.1->qiime)
  Using cached python_dateutil-2.4.2-py2.py3-none-any.whl
Collecting cycler (from matplotlib!=1.4.2,>=1.1.0->qiime)
  Using cached cycler-0.9.0-py2.py3-none-any.whl
Collecting pyparsing!=2.0.0,!=2.0.4,>=1.5.6 (from matplotlib!=1.4.2,>=1.1.0->qiime)
  Using cached pyparsing-2.0.6-py2.py3-none-any.whl
Collecting IPython (from scikit-bio<0.3.0,>=0.2.3->qiime)
  Using cached ipython-4.0.0-py2-none-any.whl
Collecting click (from biom-format<2.2.0,>=2.1.4->qiime)
  Using cached click-5.1-py2.py3-none-any.whl
Collecting pyqi (from biom-format<2.2.0,>=2.1.4->qiime)
Collecting decorator (from IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
  Using cached decorator-4.0.4-py2.py3-none-any.whl
Collecting simplegeneric>0.8 (from IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
Collecting pexpect (from IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
Collecting traitlets (from IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
  Using cached traitlets-4.0.0-py2.py3-none-any.whl
Collecting pickleshare (from IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
Collecting ptyprocess>=0.5 (from pexpect->IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
Collecting ipython-genutils (from traitlets->IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
  Using cached ipython_genutils-0.1.0-py2.py3-none-any.whl
Collecting path.py (from pickleshare->IPython->scikit-bio<0.3.0,>=0.2.3->qiime)
  Using cached path.py-8.1.2-py2.py3-none-any.whl
Installing collected packages: six, qiime-default-reference, future, burrito, pytz, python-dateutil, numpy, pandas, natsort, cycler, pyparsing, matplotlib, gdata, scipy, decorator, simplegeneric, ptyprocess, pexpect, ipython-genutils, traitlets, path.py, pickleshare, IPython, scikit-bio, cogent, pynast, click, pyqi, biom-format, burrito-fillings, qcli, emperor, qiime
Successfully installed IPython-4.0.0 biom-format-2.1.5 burrito-0.9.1 burrito-fillings-0.1.1 click-5.1 cogent-1.5.3 cycler-0.9.0 decorator-4.0.4 emperor-0.9.51 future-0.15.2 gdata-2.0.18 ipython-genutils-0.1.0 matplotlib-1.5.0 natsort-3.5.6 numpy-1.10.1 pandas-0.17.0 path.py-8.1.2 pexpect-4.0.1 pickleshare-0.5 ptyprocess-0.5 pynast-1.2.2 pyparsing-2.0.6 pyqi-0.3.2 python-dateutil-2.4.2 pytz-2015.7 qcli-0.1.1 qiime-1.9.1 qiime-default-reference-0.1.3 scikit-bio-0.2.3 scipy-0.16.1 simplegeneric-0.8.1 six-1.10.0 traitlets-4.0.0

When I try to use the package I get various errors that prove that pip hasn't really compiled any dependencies. What should I do with that? For example, let's try to import pandas

In [1]: import pandas
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-d6ac987968b6> in <module>()
----> 1 import pandas

/home/user/.conda/envs/qiime/lib/python2.7/site-packages/pandas/__init__.py in <module>()
     11                       "pandas from the source directory, you may need to run "
     12                       "'python setup.py build_ext --inplace' to build the C "
---> 13                       "extensions first.".format(module))
     14 
     15 from datetime import datetime

ImportError: C extension: /home/user/.conda/envs/qiime/lib/python2.7/site-packages/pandas/hashtable.so: undefined symbol: PyFPE_jbuf not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.

I know I can build everything manually, but I really want to fix pip.

Eli Korvigo
  • 10,265
  • 6
  • 47
  • 73
  • Why not just install qiime from conda? – Eric Dill Nov 19 '15 at 16:18
  • Either do what it tells you (and `build_ext --inplace`) or study `pandas/__init__.py` and figure out why it didn't get built automatically – Dima Tisnek Nov 19 '15 at 16:20
  • @EricDill 1. conda installation has its own problems 2. this is more about getting pip to work as it should – Eli Korvigo Nov 19 '15 at 16:32
  • @qarma this is not about `pandas` or anything, it's about pip not compiling a thing. It doesn't even compile `numpy` (or `scipy` or anything) when I try to install it directly with `pip install numpy`. – Eli Korvigo Nov 19 '15 at 16:44
  • How about any leaf package altogether, e.g. `six` and a leaf package that requires compilation, e.g. `numpy`? – Dima Tisnek Nov 19 '15 at 18:07
  • @qarma same stuff. BTW I haven't mentioned that everything works fine with easy_install, so the system itself should be fine too. – Eli Korvigo Nov 19 '15 at 19:24
  • A total WAG, is pip from virtualenv? – Dima Tisnek Nov 19 '15 at 20:21
  • @qarma it's from conda, if you mean the pip I used in my conda environment. – Eli Korvigo Nov 19 '15 at 20:47
  • @qarma Well, pip works when you provide a direct link to a github/sourceforge tarball instead of a package name, i.e. `pip install https://github.com/matplotlib/matplotlib/archive/v1.4.3.tar.gz` makes pip build the package while `pip install matplotlib==1.4.3` does not. – Eli Korvigo Nov 19 '15 at 21:28

1 Answers1

0

Passing --no-cache-dir to pip during installation seems to solve the issue, though I don't understand what caches have to do with compilation.

Eli Korvigo
  • 10,265
  • 6
  • 47
  • 73