why use django haystack slice queryset too slow?

Question

I use whoosh as the search backend.

when I get just 3 search result, code:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys
sys.path.append('/home/guomeng/projects/tapplex_ringtones')
import os
os.environ['DJANGO_SETTINGS_MODULE'] = "tapplex_ringtones.settings"

from haystack.query import SearchQuerySet
from ringtones.models import Ringtone
import time

query_word = u'sky'
t0 = time.time()
sqs = SearchQuerySet().models(Ringtone).filter(content=query_word)[:3]
t1 = time.time()
print sqs
print t1 - t0

the result is:

[<SearchResult: ringtones.ringtone (pk=u'1730')>, <SearchResult: ringtones.ringtone (pk=u'28959')>, <SearchResult: ringtones.ringtone (pk=u'25889')>]
0.422543048859> 0.422543048859

when I get all search result, code:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys
sys.path.append('/home/guomeng/projects/tapplex_ringtones')
import os
os.environ['DJANGO_SETTINGS_MODULE'] = "tapplex_ringtones.settings"

from haystack.query import SearchQuerySet
from ringtones.models import Ringtone
import time

query_word = u'sky'
t0 = time.time()
sqs = SearchQuerySet().models(Ringtone).filter(content=query_word)
t1 = time.time()
print sqs
print t1 - t0

the result is:

[<SearchResult: ringtones.ringtone (pk=u'1730')>, <SearchResult: ringtones.ringtone (pk=u'28959')>, <SearchResult: ringtones.ringtone (pk=u'25889')>, <SearchResult: ringtones.ringtone (pk=u'5303')>, <SearchResult: ringtones.ringtone (pk=u'5335')>, <SearchResult: ringtones.ringtone (pk=u'5411')>, <SearchResult: ringtones.ringtone (pk=u'1212')>, <SearchResult: ringtones.ringtone (pk=u'28473')>, <SearchResult: ringtones.ringtone (pk=u'23867')>, <SearchResult: ringtones.ringtone (pk=u'27087')>, <SearchResult: ringtones.ringtone (pk=u'26849')>, <SearchResult: ringtones.ringtone (pk=u'2973')>, <SearchResult: ringtones.ringtone (pk=u'2645')>, <SearchResult: ringtones.ringtone (pk=u'31007')>, <SearchResult: ringtones.ringtone (pk=u'11637')>, <SearchResult: ringtones.ringtone (pk=u'16957')>, <SearchResult: ringtones.ringtone (pk=u'106')>, <SearchResult: ringtones.ringtone (pk=u'2481')>, <SearchResult: ringtones.ringtone (pk=u'15697')>]
0.19460105896

why I get all the result is fast?

I am not familiar with haystack, but try use `from timeit import default_timer as timer` `start = timer()` `end = timer()` `print(end-start)` and print query, not result `print(sqs.query)` and post query here — madzohan, Nov 24 '14 at 10:38
and actually if you print both query - you will see that in the second case Django add `LIMIT` and `OFFSET` clauses https://docs.djangoproject.com/en/dev/topics/db/queries/#limiting-querysets , http://stackoverflow.com/questions/4481388/why-does-mysql-higher-limit-offset-slow-the-query-down — madzohan, Nov 24 '14 at 10:53

score 0 · Answer 1 · answered Nov 24 '14 at 10:39

Probably because:

SearchQuerySet().models(Ringtone).filter(content=query_word)[:3]

Does everything that

SearchQuerySet().models(Ringtone).filter(content=query_word)

does, but also, updates the query set with a limit clause and returns a new queryset. This could account for all or some of the time difference. To get meaningful results you'd need to average the time taken over many calls, and analyze some of the internal code - i.e. how long does the DB take to process the query, and what actually is the query etc.

why use django haystack slice queryset too slow?

1 Answers1