11

I need to do full text searching with Google App Engine. I found the project Whoosh and it works really well, as long as I use the App Engine Development Environement... When I upload my application to App Engine, I am getting the following TraceBack. For my tests, I am using the example application provided in this project. Any idea of what I am doing wrong?

<type 'exceptions.ImportError'>: cannot import name loads
Traceback (most recent call last):
  File "/base/data/home/apps/myapp/1.334374478538362709/hello.py", line 6, in <module>
    from whoosh import store
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/__init__.py", line 17, in <module>
    from whoosh.index import open_dir, create_in
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/index.py", line 31, in <module>
    from whoosh import fields, store
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/store.py", line 27, in <module>
    from whoosh import tables
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/tables.py", line 43, in <module>
    from marshal import loads

Here is the import I have in my Python file.

# Whoosh ----------------------------------------------------------------------
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'utils')))
from whoosh.fields import Schema, STORED, ID, KEYWORD, TEXT
from whoosh.index import getdatastoreindex
from whoosh.qparser import QueryParser, MultifieldParser

Thank you in advance for your help!

Martin
  • 39,309
  • 62
  • 192
  • 278

4 Answers4

7

You could probably solve your problems by downloading and using Whoosh-Appengine, the Whoosh version that's specifically targeted to working with Google App Engine.

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • 1
    I am using that version... it works only in the Development Env. – Martin Jun 23 '09 at 03:42
  • 2
    Bear in mind that even if you fix this issue, Whoosh-Appengine just uses the datastore as a file store, and doesn't respect the 1MB limit - so it may not work for any substantial amount of data. – Nick Johnson Jun 23 '09 at 09:56
  • There is now experimental support for gae inside of the current Whoosh release. See `whoosh/filedb/gae.py` in the 2.4.1 release. – floer32 Sep 03 '12 at 18:53
4

This is an official example about implementing full text search: http://code.google.com/p/guestbook-example-appengine-full-text-search/

I'm currently reading through it as I'm in the need of implementing it, maybe it will help others also.

Sam Stoelinga
  • 4,881
  • 7
  • 39
  • 54
3

The marshal module is not supported on app engine. It is there, but it is empty. That marshal is working as normal in the dev. environment has been registered as an issue.

See the documentation.

You could try the following to monkeypatch the marshal module. Put the following code before you do any other imports:

import pickle
import marshal
marshal.loads = pickle.loads
marshal.dumps = pickle.dumps # I assume it needs dumps also

I have not tried this, so I have absolutely no idea if it will work! Also be aware that pickle loads/dumps is slower than marshal loads/dumps.

codeape
  • 97,830
  • 24
  • 159
  • 188
  • In my testing for App Engine I have noticed pickle (rather than cPickle) is sometimes more than 10x slower than marshal. – gravitation Jun 29 '09 at 23:32
1

See comment #71 here: http://code.google.com/p/googleappengine/issues/detail?id=217&q=Whoosh&colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary%20Log%20Component#c71:

It's not my tool, but it says:

I have created a full text search api by porting http://whoosh.ca/ so it is avaliable on AppEngine. (it stores the index in the datastore)

You can download it from http://github.com/tallstreet/Whoosh-AppEngine/tree/master

It includes all of Whooshes features including:

1 Pythonic API. 2 Fielded indexing and search. 3 Fast indexing and retrieval 4 Pluggable scoring algorithm (including BM25F), text analysis, storage, posting format, etc. 5 Powerful query language parsed by pyparsing. 6 Pure Python spell-checker

NealWalters
  • 17,197
  • 42
  • 141
  • 251