Updating a Haystack search index with Django + Celery

Question

In my Django project I am using Celery. I switched over a command from crontab to be a periodic task and it works well but it is just calling a method on a model. Is it possible to update my Haystack index from a periodic task as well? Has anyone done this?

/manage.py update_index

That's the command to update the index from the Haystack documentation but I'm not sure how to call that from a task.

management commands should optimally just be a thin wrapper around a public API, but sadly it doesn't seem to be the case here; https://github.com/toastdriven/django-haystack/blob/master/haystack/management/commands/update_index.py — asksol, Dec 05 '10 at 12:47
as a hack you can use django.core.management.call_command("update_index"), but I would rather copy+paste the code linked above to work independently — asksol, Dec 05 '10 at 12:49
@asksol, Thanks for the reply. Why is call_command considered a hack? It seems simpler to do that the copy paste that whole command. — knuckfubuck, Dec 06 '10 at 04:37
Ah, not the whole command. Just the part that is actually doing the index, without the Django command stuff. — asksol, Dec 06 '10 at 10:19

score 30 · Accepted Answer · answered Dec 07 '10 at 00:34

30

the easiest way to do this would probably be to run the management command directly from python and run it in your task

from haystack.management.commands import update_index
update_index.Command().handle()

answered Dec 07 '10 at 00:34

Jann

1,799
3
21
38

10

this worked. Could you explain why this is better than using django.core.management.call_command("update_index") ? – knuckfubuck Dec 07 '10 at 03:26
You're right, that should work just as well, I didn't know about that function :) – Jann Dec 07 '10 at 11:49
2

Make sure to `import app.search_indexes` somewhere or it won't work I've found. The models must be registered or they will be skipped. – Gringo Suave Apr 07 '12 at 00:33
This no longer works on the 2.0.0 beta. Returns this "ImproperlyConfigured: The key 'None' isn't an available connection." – Roberto Rosario May 21 '12 at 19:25
1

You can pass command parameters like so: `update_index.Command().handle(age=1)` – Neil Jan 28 '14 at 10:32
django.core.management.call_command("update_index") throws an IO error for me in prod. However, this command works perfectly! – varagrawal May 18 '16 at 12:48
1

@Neil What does it mean age=1 - is that 1 day or hour? How to pass date from ? – Vic Nicethemer Mar 20 '17 at 08:15
my rebuild index will ask yes or no question while running in the terminal – giveJob May 25 '18 at 10:06

score 12 · Answer 2 · answered Jun 29 '12 at 11:08

12

As for version 2.0.0 beta of haystack, this code should work:

from haystack.management.commands import update_index
update_index.Command().handle(using='default')

answered Jun 29 '12 at 11:08

Wang Bin

606
4
5

4

There you can also use 'remove' option to remove non-existing entries. `update_index.Command().handle(using='default',remove=True)` – hurturk Jan 06 '13 at 09:05
6

As of haystack 2.1.0, `using` accepts list of backends. So it should be `handle(using=['default'])` – temoto Sep 11 '13 at 08:51

score 7 · Answer 3 · edited Dec 03 '15 at 04:11

7

https://github.com/django-haystack/celery-haystack

I find this package to be a great, easy plug-in app to provide haystack indexing via celery. I used it in a few projects.

edited Dec 03 '15 at 04:11

laffuste

16,287
8
84
91

answered Aug 29 '12 at 01:49

James Addison

3,086
1
17
16

score 7 · Answer 4 · answered Jul 06 '14 at 07:54

Also, since version 2 of the haystack you can call rebuild index from python as

from haystack.management.commands import update_index, rebuild_index
rebuild_index.Command().handle(interactive=False)

Where the "interactive" would prevent haystack asking question if you really want to rebuild index. This is equivalent to --no-input command line option.

If you use xapian as FTS backend please remember that multithreaded updates to index would result in DB Write Lock. So, the solution with celery-haystack package does attempt to spread index update into multiple workers (multiple thread) resulting in the lock with xapian.

Updating a Haystack search index with Django + Celery

4 Answers4

Linked