0

I can't get numpy working with Django (django-nonrel) and Google App Engine. Here's the project structure:

gae/
  virtualenv_directory/
  project/
    app/
      views.py
      algorithm.py
      ...
    lib/
      nltk/
      numpy/
      ...
    nltk_data/
    settings.py

All third-party libraries are installed in lib/ directory using command: "pip install -t . package". Also, settings.py contains the line:

sys.path.append(os.path.join(PROJECT_PATH, 'lib'))

In views.py I call a function from algorithm.py, which uses nltk. However, when calling it as a user (on development server) I get the following error:

Exception Type: ImportError
Exception Value: cannot import name multiarray
Exception Location: /home/me/gae/project/lib/numpy/core/init.py in , line 6
Python Executable: /home/me/gae/virtualenv_directory/bin/python
Python Version: 2.7.3

Traceback ends with line:

File "/home/me/gae/project/lib/numpy/core/init.py" in
6. from . import multiarray

Local vars:

file None
absolute_import None
package None
path None
name None
version None
doc None
print_function None

Ultimate answer to relative python imports says that:

Relative imports use a module's name attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to 'main') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.

So it seems that the error is caused by name equal to 'None'. But what is the reason for it?

In ./manage.py shell I can import numpy.core.multiarray or call the function from algorithm.py without any errors.

I've also tried to use GAE numpy, adding the following lines to app.yaml:

- name: numpy
  version: latest

But the result was another import error, that is:

Exception Value: cannot import name scimath
Exception Location: /home/me/gae/virtualenv_directory/local/lib/python2.7/site-packages/numpy/lib/init.py in , line 17
Python Executable: /home/me/gae/virtualenv_directory/bin/python

/home/me/gae/virtualenv_directory/local/lib/python2.7/site-packages/numpy/lib/init.py in
17. from . import scimath as emath

Once again, all local vars mentioned above are equal to 'None'.

Any suggestions how to fix this?

Community
  • 1
  • 1
cafe_
  • 213
  • 3
  • 9
  • Are you trying to run python with virtualenv activated. This does not normally work. – Tim Hoffman Nov 11 '13 at 02:53
  • Also consider using appengine_config.py for doing path manipulation rather than using your settings.py - this is run by the server before doing anything with your code. https://developers.google.com/appengine/docs/python/tools/appengineconfig – Tim Hoffman Nov 11 '13 at 02:56
  • Also watch where you put your local copy of numpy you don't want to deploy it. You will run the production copy that is already there. – Tim Hoffman Nov 11 '13 at 03:00
  • 1) Yes, virtualenv is activated when I run development server. However, deactivation results in the same error (cannot import multiarray). 2) I'm going to try appengine_config.py 3) Could you elaborate a little more on this? I have 3 copies of numpy - one in lib/ directory, one in virtualenv_directory and one global, installed without pip. – cafe_ Nov 11 '13 at 22:59
  • The only version of numpy that the SDK can access is the one in your lib directory. This is what my appengine config looks like http://pastebin.com/DvrDZxN6 . You probably should check that the version in your lib dir can actually be used, in a python shell. – Tim Hoffman Nov 11 '13 at 23:57
  • 1) Adding 'import numpy' to views.py results in the same error (multiarray) when trying to load main page. 2) I can import library from lib/ directory. 3) When 'global' version of numpy (i.e. the one installed in /usr/local/lib/python2.7/dist-packages) is removed, virtualenv is deactivated and I append '/home/me/gae/project/lib' to sys.path in python shell, it is possible to import numpy or any other part of it. 4) Numpy version is 1.8.0, if it does matter at all. – cafe_ Nov 12 '13 at 11:57
  • I still don't have any idea how to fix it. Here's the log from Django: http://pastebin.com/Rhz05Vnr – cafe_ Dec 03 '13 at 17:45
  • Well, it works if I remove all template tags from template. '''import numpy''' stays in views.py, and I really doubt they (views.py) are called/imported when url uses direct_to_template. It's stunning. – cafe_ Dec 03 '13 at 21:45
  • I've downloaded django-nonrel testapp from https://github.com/django-nonrel/django-testapp, created another virtualenv, installed numpy once again with pip(1.8.0) and imported numpy in views.py. It still doesn't work: `cannot import name _umath_linalg` – cafe_ Dec 03 '13 at 23:27
  • 1
    Wow! It works - but I don't understand why. First, I removed numpy from lib/. Second, I've installed numpy 1.6.1 with command `sudo pip install -t . numpy==1.6.1` (being in aforementioned lib directory). The last thing was to add numpy 1.6.1 to app.yaml. I think it was some compilation-specific failure or missing user permissions (but why did it succeed during compilation?). Anyway, thanks for help. – cafe_ Dec 03 '13 at 23:46

1 Answers1

2

I had the same problem on Mac. You can't install numpy 1.6.1 on Mac despite using

CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install numpy==1.6.1

And I couldn't use the latest numpy due dependencies on multiarray namespace. Besides currently GAE only supports only up to numpy===1.6.1 hence it makes sense to have any higher version installed.

The solution was for me to install 1.6.2, which I could install on Mac and has still the needed multiarray. Now I can import numpy in GAE and it works.

Houman
  • 64,245
  • 87
  • 278
  • 460