Elastic Search Reindex : Wait for completion

Question

I'm trying to reindex 2695140 documents, using Nest C#. I need to calculate how much time it taken to reindex all the documents, for which I've written the logs. But after running for 1 minute, my code is returning an invalid response (Failed) but the documents are getting indexed properly as we have triggered Reindex endoint of elastic search.

I would want my code should wait until the reindex operation is completed so that I can calculate the total time taken to reindex. Below is the code I'm using

return await Client.ReindexOnServerAsync(selector => selector
                                .Source(src => src
                                  .Index(_config.SomeIndex))
                               .Destination(dest => dest
                                  .Index(newIndexName).OpType(OpType.Index))
                               .WaitForCompletion(true));

Thanks in advance.

score 7 · Answer 1 · answered Sep 17 '20 at 09:55

I would want my code should wait until the reindex operation is completed

I don't know which programming language are you using but essentially for languages following "One Thread per Request" model it is not wise to wait for the reindex operation. The time taken by the operation will be proportional to the number of documents to re-index and it blocks the thread (consuming resource) until the operation is complete.

Instead you should:

Re-index without waiting for completion e.g :

POST _reindex?wait_for_completion=false
   {
    "source":{
        "index":"book"
    },
    "dest":{
        "index":"book_new1"
    }
}

Response: will have task_id

Use tasks API for tracking the completion of task. It will also contain status whether the request succeeded or not and the time taken by the operation. Sample response for tasks API will look like :

{
  "completed" : true,
  "task" : {
    "node" : "jF8smI1eR1mwwNxl8_7z2A",
    "id" : 2427911
    },
    "description" : "reindex from [book] to [book_new1][_doc]",
    "start_time_in_millis" : 1600335207787,
    "running_time_in_nanos" : 640430472,
    "cancellable" : true,
    "headers" : { }
  },
  "response" : {
    "took" : 634,  // <====== Time taken by operation
    "timed_out" : false,
    "total" : 3,
    "updated" : 0,
    "created" : 3,
    "deleted" : 0,
    "batches" : 1,
    "version_conflicts" : 0,
    "noops" : 0
  }
}

You can periodically check (using cron/scheduler/etc) until completion and take required action.

Thanks @Sahil Gupta . It might help me for getting the execution time. I would have to update my logic to get the ID and check it with another request. — Moulali Shaik, Sep 17 '20 at 10:36
Can you please accept it as answer and upvote if it helped you. — Sahil Gupta, Sep 17 '20 at 10:53
I'm still waiting for something which I should not hit the logic to check the execution time based on the ID at later point of time. My function has nothing else to do than this one operation. WaitForCompletion(true)) is not waiting more than a minute, I want my logic wait until the operation finishes. — Moulali Shaik, Sep 24 '20 at 05:26
Good idea to use `$URL/_tasks({ actions: '*reindex' })` to know if there are running reindex tasks and check periodically before running a new task like a delete, another reindex,... — vintagexav, Jun 06 '23 at 09:32

score 3 · Answer 2 · answered Jul 14 '21 at 08:52

In addition to the @sahil-gupta answer please checkout this option wait for completion - after starting an asynchronous task you can wait for a specific task or even all started tasks to be completed using another request:

curl "http://127.0.0.1:9200/_tasks/?wait_for_completion=true&timeout=100s"

Please note that except of a standard timeout response (json, related to the timeout query parameter) it may end up with an error like this:

context deadline exceeded (Client.Timeout exceeded while awaiting headers)

so it is needed to handle possible-long-tasks with a simple loop of such requests.

Elastic Search Reindex : Wait for completion

2 Answers2