0

We're getting the exception below when performing an update-by-query via Elastic's NEST api.

We understand the exception relates to the memory allocation of Elasticsearch's java heap but we would like to explore ways to resolve the issue without having to increase the size of the heap.

How can we reduce the size of the memory required to perform the update?

index: ServerError: 429Type:
es rejected execution exception Reason: rejected execution of coordinating operation
[coordinating and primary bytes=792940313, replica bytes=0, all bytes=792940313,
coordinating_ operation bytes=162549881.
max coordinating and primary bytes=858993459"

EDIT 1: Added the NEST execution below and answered questions:

The number of documents will depend on our customer's data profile but is in the region of 10k to 100k of documents.

We're using Slice(1)

var resp = _elasticClient.UpdateByQuery<ServiceModel.ElasticSearch.Matter>(q => q
            .Index(indices)
            .Script(script)
            .Refresh(true)
            .ScrollSize(1000)
            .Slices(1)
            .Query(query => query
                .Nested(n => n
                    .Path(p => p.MatterTypes)
                    .Query(q2 => q2
                        .Bool(b => b
                            .Must(mu => mu
                                .Term(mtid => mtid
                                    .Field(f => f.MatterTypes.First().Id)
                                    .Value(request.Id)
            ))))))
        );

And here's the script in case there's any issues here..

var script = $@"
    int matterTypeId={request.Id};
    int[] removePracticeAreaIds = new int[{request.Remove.Count}];
    int[] addPracticeAreaIds = new int[{request.Add.Count}];
    {assignRemovePracticeAreaIdsArray}
    {assignAddPracticeAreaIdsArray}
    for (int i=0;i<ctx._source.matterTypes.size();i++)
    {{
        if (ctx._source.matterTypes[i].id == matterTypeId)
        {{
            if (removePracticeAreaIds.length > 0)
            {{
                for (int k=0; k<removePracticeAreaIds.length; k++)
                {{
                    for (int j=ctx._source.matterTypes[i].practiceAreas.size()-1; j>=0; j--)
                    {{
                        if (ctx._source.matterTypes[i].practiceAreas[j] == removePracticeAreaIds[k])
                        {{
                            ctx._source.matterTypes[i].practiceAreas.remove(j);
                        }}
                    }}
                }}
            }}

            if (addPracticeAreaIds.length > 0)
            {{
                for (int k=0; k<addPracticeAreaIds.length; k++)
                {{
                    if(!ctx._source.matterTypes[i].practiceAreas.contains(addPracticeAreaIds[k]))
                    {{
                        ctx._source.matterTypes[i].practiceAreas.add(addPracticeAreaIds[k]);
                    }}
                }}
            }}

            if(ctx._source.matterTypes[i].practiceAreas.size() == 0)
            {{
                ctx._source.matterTypes[i].practiceAreas.add({Common.Consts.Defaults.UnmappedLookup});
            }}
        }} 
    }}";
Drammy
  • 940
  • 12
  • 30
  • what does the request you look send look like? how many documents is it likely to udpate? – warkolm Aug 10 '22 at 08:40
  • You should try running [with `slice=1`](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-slice) (or 2/3/4, your mileage may vary depending on how many primary shards your index has) instead of `slice=auto` (default) which will consume less memory. – Val Aug 10 '22 at 08:41
  • @MarkWalkom Hopefully the detail I've added explains things better – Drammy Aug 11 '22 at 10:02
  • tldr if trying different slice sizes doesn't work, as Val mentions above, you will probably need to increase heap – warkolm Aug 12 '22 at 00:40
  • @MarkWalkom OK, thanks - then I think we'll need to do some data volume sizing, shard count adjusting, slice count adjustments and potentially adjust the heap size. – Drammy Aug 12 '22 at 11:35

0 Answers0