0

I'm indexing a bunch of documents using Elastic Search's Bulk API from the Javascript (NodeJS) client. I'm sending a thousand docs with each call. The instance handles it until before it reaches 100 calls (100K documents, approx.). Then it goes down returning a Service Unavailable (503) error.

Before doing a new call, I wait for the previous to complete and an extra second.

Searching on this matter, I found a post that talks about a fix for Rails: https://medium.com/@thetron/dealing-with-503-errors-when-testing-elasticsearch-integration-in-rails-ec7a5f828274. The author uses the following code to make the errors go away:

before do
    repository.create_index!
    repository.client.cluster.health wait_for_status: ‘yellow’
end

Based on that, I wrote the following:

const body = [
    // 1K actions/docs
];
elastic.cluster.health({
    waitForStatus: 'yellow',
    timeout: '60s', // I also tried using the default timeout
    requestTimeout: 60000
}, function (error, response) {
    if (!!error) {
        console.error(error);
        return;
    }
    elastic.bulk({
        body: body
    }, function (error, response) {
        if (!!error) {
            console.error(error);
            return;
        }
        console.log('Success!');
    });
});

Not sure if it makes any difference, but the instance is running on AWS. Due the big number of docs, maybe scaling up the instance is a solution. But I wanted to figure out how to handle this error before going that way. Even if I have to make my code somewhat slower.

Gustavo Straube
  • 3,744
  • 6
  • 39
  • 62

1 Answers1

1

Your best bet is to scale up your cluster as you've said, but your loading process should be able to handle failures as well.

That being said, the following are the Elasticsearch cluster statuses:

  • red - There's at least one unallocated primary shard
  • yellow - There's at least one unallocated replica shard
  • green - Everything is allocated and healthy

So in you example above, you don't want to wait for yellow, you want to wait for green.

Alcanzar
  • 16,985
  • 6
  • 42
  • 59
  • I think my instance is never green. Thinking of that, probably scaling up is the only solution. – Gustavo Straube Aug 26 '16 at 14:17
  • If your instance is never green then you don't have enough nodes for replicas to be on different nodes than primary shards. You can set the replicas to 0, but then you have no redundancy. – Alcanzar Aug 26 '16 at 14:18
  • You can diagnose the reason you aren't getting to green using one of my other answers: http://stackoverflow.com/a/23816954/2785358 – Alcanzar Aug 26 '16 at 14:20
  • Thanks for the details. I'll check that answer of yours. – Gustavo Straube Aug 26 '16 at 14:30