Need external version support in OpenSearch Update operation

Question

I am using OpenSearch to index JSON documents & make them searchable. All documents have update timestamp field in EPOCH format. The problem is I can get update request where document body contains an older update time. My application should skip the update if the current document update time is older than the update time field in existing document stored in OpenSearch

To fulfil the requirement, I added external version in HTTP request /test_index/_update/123?version=1674576432910&version_type=external.

But I am getting error

Validation Failed: 1: internal versioning can not be used for optimistic concurrency control. Please use if_seq_no and if_primary_term instead

I read about if_seq_no & if_primary_term fields. They can't be used to solve my problem. Has anyone else encountered this problem & solved it? Please share. Or if anyone know about any plugin that I can install to support this, please share.

score 0 · Answer 1 · answered Jan 24 '23 at 16:21

You should use the "if_seq_no" and "if_primary_term" parameters to perform optimistic concurrency control.

To solve your problem, you could first retrieve the existing document from OpenSearch using the document ID, and check the update timestamp field. If the existing timestamp is newer than the one in the update request, you can skip the update. Otherwise, you can include the "if_seq_no" and "if_primary_term" parameters in your update request, along with the updated document. The "if_seq_no" parameter should be set to the sequence number of the existing document, and the "if_primary_term" parameter should be set to the primary term of the existing document.

You can use the Update API for this... or the Optimistic Concurrency Control (OCC) mechanism, which is based on a combination of _seq_no and _primary_term fields.

Thank you for the answer, I thought it. One problem is there. Example: I added if_seq_no & if_primary_term in the update request. Before the actual update happens, I got another update request for same document with higher update timestamp. Second update should also happen. But seq_no will change due to first update & second update will fail because of that. I don't have control & simultaneous update calls can come & we can miss valid update. — muthuk, Jan 25 '23 at 10:10

score 0 · Accepted Answer · answered Jan 27 '23 at 04:56

Sadly neither OpenSearch nor ElasticSearch supports external version in update request. And I don't see the feature getting added in near future. You can solve your specific problem using scripting. OpenSearch supports multiple scripting languages including Painless script. You can write a script that will compare a specific field (in your case update timestamp). And if condition is true, it will go ahead & update the fields with the new values.

{
    "script": {
        "lang": "painless",
        "source": "if (params.updateTimestamp > ctx._source.updateTimestamp) {for (entry in params.entrySet()) {ctx._source[entry.getKey()] = entry.getValue();}}"
    }
}

You can see a sample script above which will silently skip any update if new document has older timestamp. You can even throw exception also & handle it from you application. That way you can track number of requests with such issue.

You can use a similar script as stored script & use it in your update request. You can get more details including sample HTTP request & Java code in this article.

Need external version support in OpenSearch Update operation

2 Answers2