Django Haystack-Xapian search fails with special characters and spaces

Question

I'm working on an existing Django project which uses haystack with xapian backend for a global search feature. However the search seems to fail while I search with some special characters like & and spaces etc. I tried but could not find out a way to fix it.

Is there a way I can escape these characters and make the search work? I'm using PostgreSQL at the back-end. Any pointers will be very helpful.

Update: Search feature uses a SearchForm and the query string comes in q field of the from. This is used as below in the code.

sqs = self.searchqueryset.auto_query(self.cleaned_data['q'])

I've tried using:

sqs = self.searchqueryset.filter(self.cleaned_data['q'])

and

sqs = self.searchqueryset.filter(content=Clean(self.cleaned_data['q']))

with no luck. I'm still not able to search with spaces and special characters like &.

Have you seen this? http://stackoverflow.com/questions/12961215/escaping-ampersand-character-in-sql-string — yuvi, Mar 15 '14 at 20:04
@MatthewWilcoxson I'm not getting any error messages. It's just that I'm not getting any results while I'm searching with special characters like '&' in my search string. It works otherwise. — Ankit Jaiswal, Mar 16 '14 at 06:32
Have you managed to run the SQL query and get results (without django that is)? — yuvi, Mar 16 '14 at 14:32

score 1 · Answer 1 · answered Mar 15 '14 at 21:25

1

Escaping depends on how you are using Haystack, however, the haystack.inputs.Clean class may be what you are looking for. Just pass in whatever you are searching for:

q = "amp & sand"
q_clean = haystack.inputs.Clean(q)
sqs = SearchQuerySet().filter(content=q_clean)

See the docs here: http://django-haystack.readthedocs.org/en/latest/inputtypes.html#clean

answered Mar 15 '14 at 21:25

Matthew Wilcoxson

3,432
1
43
48

I've already tried it, but it is still not fetching the data as per the query result if there are special characters like '&' in the search string. – Ankit Jaiswal Mar 16 '14 at 06:33

Jorge Leitao · Answer 2 · 2014-06-15T07:13:32.280

Disclaimer: I'm the (new) maintainer of Xapian-Haystack.

I believe that what you are reporting is related with how Xapian-Haystack parses the string into indexable terms for searching: because it uses Xapian default generator, which ignores special characters, you are unable to search special characters because they are not even indexed.

For instance, the string "^ best-seller" is indexed as: "best" and "seller": an search on "^" or an exact search on "^ best-seller" will fail.

This has been solved for now: all special characters should now be supported.

Space is never indexed because in Xapian-Haystack (or Haystack) since it is the word separator.

While I wouldn't recommend using master on production at the moment, maybe this answers your question.

yeah I understand that spaces are not indexed and that makes sense. However, searching the special characters was important in my case. After trying and spending a good amount of time, I had to replace xapian with another search engine :( — Ankit Jaiswal, May 24 '14 at 08:33

Django Haystack-Xapian search fails with special characters and spaces

2 Answers2