2

My issue started identical to this one: Python executable not finding libpython shared library

I updated .bashrc with export LD_LIBRARY_PATH=$HOME/local/lib/python/2.7.6/lib and things were fine. Python works, and I installed pip. But now, I'm running into something similar when installing cython with pip. I get this error message when I execute pip install cython:

gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip_build/cython/Cython/Plex/Scanners.o -L. -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Plex/Scanners.so

/usr/bin/ld: cannot find -lpython2.7

collect2: ld returned 1 exit status

error: command 'gcc' failed with exit status 1

I cannot add $HOME/local/lib/python/2.7.6/lib to /etc/ld.so.conf and run ldconfig as I do not have root. I was under the impression that setting the LD_LIBRARY_PATH was the way around this, but this appears to not be true for compilation. Is there a way to get the compiler to see this local library without running root commands?

Community
  • 1
  • 1
Matt Hancock
  • 3,870
  • 4
  • 30
  • 44

2 Answers2

3

Update:

The LD_LIBRARY_PATH is only used by the dynamic loader at runtime, not at build time, so that is not the issue. The issue is that you forgot to put the -L/path/to/pylib before the -l. I've never had to use LIBRARY_PATH because a build requires path extension that is specific to a given build, so you never set LIBRARY_PATH you just use -L. You would only set if if you are going to regularly do builds that use a specific library, and even then I find it better to use -L because sooner or later this will cause linker to find the wrong lib and by then you will have forgotten that it's because LIBRARY_PATH is set permanently.

There are many ways to set -L values in a build: if you run the compiler from command line you don't need that env var, you just specify as many -L as required as part of the command; if you use a makefile, you edit whatever make variable you are using, such as CFLAGS or other, different platforms have different conventions. So whereas setting -L directly will always work, setting CFLAGS will only work if that is the variable used by the makefile.

Now this is a python installation so where to set this may not be obvious, but I am sure there is another way than setting LIBRARY_PATH. In principle any python package you install, if it involves compilation of C++ modules, could require edit of the setup.py to set library paths. For example 

Extension(...,
          library_dirs=['/usr/X11R6/lib'],
          ...)

Since you mention nympy, another place to set this might be in site.cfg (see Supplying NumPy site.cfg arguments to pip).

Old (wrong) answer:

Set your LD_LIBRARY_PATH in your bash console. If this doesn't work then it's because you have the wrong path: check by echoing the environment var.

Once you get that to work, edit your .bashrc or .profile then exit your shell and restart it. Echo the env var to verify that contains the part you added.

Also, ensure that you are appending to the path rather overwriting it:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/...
export LD_LIBRARY_PATH

Because python lib might depend on .so in other folders, if the linker can't find them it may appear as though it is the python lib that was not found. This is not explained on the page you linked to in your question.

Community
  • 1
  • 1
Oliver
  • 27,510
  • 9
  • 72
  • 103
  • I tried the things you mentioned with no luck. I was able to get cython to build by running `CFLAGS="...pypath.../lib" python setup.py install`. It didn't feel right, but at least it built. However, when I try building numpy with a similar trick, I run into the same issue but with gfortran. Specifying both CFLAG and FFLAG doesn't work. Something is not right here. – Matt Hancock Apr 28 '14 at 00:02
  • At your update: yes that's why my previous comment allowed cython to compile. I actually meant to put CFLAGS="-L...pypath.../lib". This was more of a workaround, as it did not work with numpy installation as I mentioned. Setting `LIBRARY_PATH` a proper way to make the compiler aware of `libpython2.7.so` in the `...pythonpath.../lib` dir. I appreciate your taking the time to look at this though. – Matt Hancock Apr 28 '14 at 01:36
  • @ChesterVonWinchester Well, I'd like to upvote your answer because it contained a critical clue, but I can't because it is not the answer. The answer is to use -L (as apparently you were implying but I didn't get that), how to use it will depend on your specifics, as I have explained in my updated answer. – Oliver Apr 28 '14 at 12:18
  • For my case, setting `LIBRARY_PATH` was a better solution. Setting the `-L` option was not as simple adding this flag in a makefile as I'm running python setup.py scripts. Even if I can append it in all the proper places in the setup.py file for a particular package, this probably won't work for all packages forcing me to tinker with the setup.py and compile flags for every new package (see cython vs numpy in the first comment). I see setting `LIBRARY_PATH` as a local version of `ldconfig` which was exactly what I needed since I don't have root. – Matt Hancock Apr 28 '14 at 13:00
  • All that aside, I marked yours as answer since this question turned into more of a question on compile time linking than about cython or python, and your answer provides better advice for the general case in that respect. – Matt Hancock Apr 28 '14 at 13:02
  • @ChesterVonWinchester OK, I can see your point. Still I am sure there is another way than setting LIBRARY_PATH system-wide, I've never had to set that. I've updated, if you try one of the updates, please comment. – Oliver Apr 28 '14 at 20:36
0

OK after some more digging I found this: LD_LIBRARY_PATH vs LIBRARY_PATH

Setting LIBRARY_PATH to the same path as LD_LIBRARY_PATH made the compiler aware of the python lib. cython/numpy/scipy all built and installed no problem afterwords.

Community
  • 1
  • 1
Matt Hancock
  • 3,870
  • 4
  • 30
  • 44
  • Ah duh of course now I see the issue, I've updated my answer hopefully that works (I've never had to set that env var so please try the update). – Oliver Apr 28 '14 at 01:19