I have an AWS OpenSearch cluster and I'm working on a rotation system. There is an alias that contains 2 indices, 1 for previous month and 1 for the current month. At the start of a new month I want to store the event data of the previous month to a local NAS and delete that index. In this way I will always have the data in the OpenSearch for a max of 2 months.
I am having trouble with the script that backups all the events. I am using the scroll API to get the events in batches and write it to the NAS. This works well on my test data. But my production data has over 100 million events and getting this data per batch of 10000 gives me a 429 Too Many Requests error after about an hour and a half (about 2000 requests).
Is there a way to get this solved/work arround it, or am I tackling this issue the wrong way?
