We're getting the exception below when performing an update-by-query via Elastic's NEST api.
We understand the exception relates to the memory allocation of Elasticsearch's java heap but we would like to explore ways to resolve the issue without having to increase the size of the heap.
How can we reduce the size of the memory required to perform the update?
index: ServerError: 429Type:
es rejected execution exception Reason: rejected execution of coordinating operation
[coordinating and primary bytes=792940313, replica bytes=0, all bytes=792940313,
coordinating_ operation bytes=162549881.
max coordinating and primary bytes=858993459"
EDIT 1: Added the NEST execution below and answered questions:
The number of documents will depend on our customer's data profile but is in the region of 10k to 100k of documents.
We're using Slice(1)
var resp = _elasticClient.UpdateByQuery<ServiceModel.ElasticSearch.Matter>(q => q
.Index(indices)
.Script(script)
.Refresh(true)
.ScrollSize(1000)
.Slices(1)
.Query(query => query
.Nested(n => n
.Path(p => p.MatterTypes)
.Query(q2 => q2
.Bool(b => b
.Must(mu => mu
.Term(mtid => mtid
.Field(f => f.MatterTypes.First().Id)
.Value(request.Id)
))))))
);
And here's the script in case there's any issues here..
var script = $@"
int matterTypeId={request.Id};
int[] removePracticeAreaIds = new int[{request.Remove.Count}];
int[] addPracticeAreaIds = new int[{request.Add.Count}];
{assignRemovePracticeAreaIdsArray}
{assignAddPracticeAreaIdsArray}
for (int i=0;i<ctx._source.matterTypes.size();i++)
{{
if (ctx._source.matterTypes[i].id == matterTypeId)
{{
if (removePracticeAreaIds.length > 0)
{{
for (int k=0; k<removePracticeAreaIds.length; k++)
{{
for (int j=ctx._source.matterTypes[i].practiceAreas.size()-1; j>=0; j--)
{{
if (ctx._source.matterTypes[i].practiceAreas[j] == removePracticeAreaIds[k])
{{
ctx._source.matterTypes[i].practiceAreas.remove(j);
}}
}}
}}
}}
if (addPracticeAreaIds.length > 0)
{{
for (int k=0; k<addPracticeAreaIds.length; k++)
{{
if(!ctx._source.matterTypes[i].practiceAreas.contains(addPracticeAreaIds[k]))
{{
ctx._source.matterTypes[i].practiceAreas.add(addPracticeAreaIds[k]);
}}
}}
}}
if(ctx._source.matterTypes[i].practiceAreas.size() == 0)
{{
ctx._source.matterTypes[i].practiceAreas.add({Common.Consts.Defaults.UnmappedLookup});
}}
}}
}}";