Python line_profiler not finding module

Question

I recognize that this is an installation failure on my part, and I'm sorry to lay this uninteresting and inconsequential question at your feet, but for the life of me I can't manage to figure out what is going wrong and I've run out of ideas. I'm hoping someone will be able to quickly point out the obvious.

I am trying to profile a python script (using Kern's line_profiler), and the script needs to load the netCDF4 module. I have installed both line_profiler and netCDF4 with pip. Both are reported as present and updated when I queue pip for the list of installed packages.

Without using the profiler, my script runs without problems, meaning that the netCDF4 module is loaded properly. However, if I run 'kernprof -l -v myscript.py' from the "myscript" directory, I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/kernprof", line 9, in <module>
   load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
  File "Library/Python/2.7/site-packages/kernprof.py", line 221, in main
   execfile(script_file, ns, ns)
  File "myscript.py", line 5, in <module>
   from netCDF4 import Dataset
ImportError: No module named netCDF4

I am running Python from an installation at /opt/local/bin/python, which is listed first in my PATH.

So, in any case, if the default Python version that I have set is the same as that which appears first in my PATH, and that default version is able to access the netCDF4 module, why isn't line_profiler?

can you import some other standard module `import os` and then `print(os.__file__)` to see where is the standard library located when running the profiler? — Jean-François Fabre, Jan 03 '17 at 20:44
print (os._file_) gives: `AttributeError: 'module' object has no attribute '_file_'` — Nordlendingen, Jan 03 '17 at 21:12
okay, wrong pick, built-in module :) :try `import csv; print(csv.__file__)`. Also: `import sys; print(sys.executable)`. Both results would be helpful. — Jean-François Fabre, Jan 03 '17 at 21:26
`print (csv._file_)` also gave the same attribute error, but `print (sys.executable)` is illuminating. Running without the profiler it yields `/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python`, whereas with the profiler it yields `/usr/bin/python`. So I guess the next question, is there a trick to force line_profiler to run under the Python version I have set as 'default'? — Nordlendingen, Jan 03 '17 at 21:47
BTW it is `__file__` with double underscores. Not single. Hence the error... — Jean-François Fabre, Jan 03 '17 at 21:58
can you post `/usr/local/bin/kernprof` contents if not too big? — Jean-François Fabre, Jan 03 '17 at 22:03
Ah! Thanks! Here is the contents from /usr/local/bin/kernprof: `#!/usr/bin/python # EASY-INSTALL-ENTRY-SCRIPT: 'line-profiler==1.0','console_scripts','kernprof' __requires__ = 'line-profiler==1.0' import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.exit( load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')() )` — Nordlendingen, Jan 03 '17 at 22:18
can you check the first lines of `myscript.py` as well? and see my answer. — Jean-François Fabre, Jan 03 '17 at 22:21
Yes, running `print(os.__file__), print (csv.__file__) and print (sys.executable)` with the proper double underscores now yields (without profiler): `/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.pyc /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python` — Nordlendingen, Jan 03 '17 at 22:24
and with profiler: `/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.pyc /usr/bin/python` — Nordlendingen, Jan 03 '17 at 22:25

Jean-François Fabre · Accepted Answer · 2017-01-03T22:40:59.930

kernprof has a shebang that redirects to the default python install which doesn't have all the required modules.

You can force the use of your "complete" python install by doing:

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/‌Resources/Python.app‌/Contents/MacOS/Pyth‌on /usr/local/bin/kernprof -l -v myscript.py

So shebang is ignored, and you run the profiler with the version of python containing all the required packages.

Python line_profiler not finding module

1 Answers1