0

I installed guppy the memory profiler from its svn#95 via "sudo python setup.py install".

It looks properly installed.


yey@yey:/usr/local/lib/python2.7/dist-packages/guppy/heapy$ ls *.so *.py
AbstractAlgebra.py  ImpSet.py          Path.py         Remote.py  Use.py
Classifiers.py      __init__.py        pbhelp.py       RM.py      View.py
Console.py      Monitor.py         Prof.py         Spec.py
Doc.py          OutputHandling.py  RefPat.py       Target.py
heapyc.so       Part.py        RemoteConstants.py  UniSet.py

But I still can't import it. Guppy's Python source does this import so it should succeed.


>>> import guppy.heapy
>>> import guppy.heapy.heapyc
# trying /usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.so
# trying /usr/local/lib/python2.7/dist-packages/guppy/heapy/heapycmodule.so
# trying /usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.py
# trying /usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.pyc
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named heapyc

My question is, Python clearly made an attempt to import the file at the correct location. Why did it fail? Is it because the .so file corrupted? Or is it my ld.so.cache bad somehow? Thanks!

Yey
  • 554
  • 3
  • 15
  • 1
    Yes, it could be corrupted, or empty, or a perfectly valid ELF shared object but for for the wrong architecture or platform, or… If you want to test it from within Python, you can try, e.g., `import ctypes; ctypes.CDLL('/usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.so')`, which will give you a traceback with more information, including the `dlerror` output, which may help. – abarnert Sep 25 '13 at 21:39
  • Thanks a lot! It turned out to be just wrong permission. I wish I can file a bug to Python on better error msg. Also it's kinda strange that setup.py made all *.py readable to everyone but not *.so `cdll.LoadLibrary('/usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.so') OSError: /usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.so: cannot open shared object file: Permission denied` – Yey Sep 26 '13 at 00:05
  • You should post your own answer (ideally what the problem was, how you discovered it, and how you fixed it) and accept it, for the benefit of future readers. (If you think I did most of the work, I can write an answer instead, but really, I think it belongs to you here.) – abarnert Sep 26 '13 at 00:31
  • I want to but there is still one missing link in my head. I found some posts about LD_LIBRARY_PATH. I wonder why it worked for me without setting that. Is python27 just smarter?Thanks! [link](http://stackoverflow.com/questions/1099981/why-cant-python-find-shared-objects-that-are-in-directories-in-sys-path?rq=1) – Yey Sep 26 '13 at 17:16
  • Setting `LD_LIBRARY_PATH` is how you add additional search paths. Python doesn't just open `heapyc.so` without a path; it uses an absolute pathname, so the search paths are irrelevant there. If, say, `heapyc.so` had a dependency on `libguppysupport.so`, and that file was actually in `/usr/local/share/guppy/libs/libguppysupport.so`, the dynamic loader wouldn't be able to find it unless you added `/usr/local/share/guppy/libs` to `LD_LIBRARY_PATHS`, but if all of its dependencies are in normal locations (or use absolute paths), that's not necessary either. So, usually, it won't come up. – abarnert Sep 26 '13 at 19:25
  • I see. Pls post ur comment as an answer. Your pointer on how to troubleshoot was the key. Thanks. – Yey Sep 27 '13 at 17:35
  • 1
    Done. By the way, you _could_ file a bug on this, but it would be closed as "already fixed", because the new importer in 3.1 already gives you better error messages. (Well, I haven't tested whether 3.1 does or not, but 3.3 does, and it's the importer rewrite in 3.1 that made such error messages possible.) – abarnert Sep 30 '13 at 18:38

1 Answers1

1

There are many possible problems with the .so file that could cause this—no read access, a corrupted file, an empty file, a perfectly valid library but for the wrong platform/architecture, etc. Worse, the .so itself may be fine, but it may have load-time dependencies on a different file that has any of the above problems.

Unfortunately, the Python 2.x importer doesn't show you which problem it's actually hit; all you can tell is that, for some reason, the call to open the shared library failed.

It's worth noting that in 3.1 or later, you would have gotten a much more useful error message, something like this:

ImportError: dlopen(/usr/local/lib/python3.3/dist-packages/guppy/heapy/heapyc.so, 2): no suitable image found.  Did find:
    /usr/local/lib/python3.3/dist-packages/guppy/heapy/heapyc.so: Permission denied

However, that's only possible because the importer was rewritten from scratch for 3.1, and there's no way such a radical change is ever going to be backported to 2.7.

Most platforms come with tools that let you test shared libraries, and this is really the best way to diagnose the problem.

But for a simple and platform-independent test, you can just use the ctypes library that comes with Python itself:

>>> import ctypes
>>> ctypes.CDLL('/usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.so')

You should get an error, like this:

OSError: /usr/local/lib/python2.7/dist-packages/guppy/heapy/heapyc.so: cannot open shared object file: Permission denied

In this case, the file isn't readable (or, on platforms that require shared libraries to be executable, it either isn't readable or isn't executable), which should be enough to fix the problem. So, a chmod a+r should fix it (although you may want to go further and figure out why it wasn't readable in the first place).

If the error doesn't tell you enough to fix it yourself, and searching doesn't help, at least you can come to SO and ask a question that will be much more likely to get an immediate answer…

abarnert
  • 354,177
  • 51
  • 601
  • 671