Migration of data from one elastic search index to another in different region in AWS using manual snapshots

Question

I have created two elastic search domains - one in us-east-1 and another in us-west-2. I have registered manual snapshot repository in us-east-1 domain and have taken snapshot and the data is in s3 bucket in us-east-1.

How should I go about doing the restoration?

Main questions:

1. Do I have to do cross-region replication of the s3 bucket to us-west-2, so that everytime a snapshot is taken in us-east-1, it automatically reflects to us-west-2 bucket?

2. If so, do I have to be in us-west-2 to register manual snapshot repository on the domain and that s3 bucket?

3. Will the restore API look like this? curl -XPOST 'elasticsearch-domain-endpoint-us-west-2/_snapshot/repository-name/snapshot-name/_restore'

Answer 1

1. You don't need to create S3 buckets in several regions. Only one is sufficient. So your S3 repository will be in us-west-2

2. You need to create the snapshot repository in both of your clusters so that you can access it from both sides. From one cluster you will create snapshots and from the second cluster you'll be able to restore those snapshots.

3. Yes, that's correct.

Answer 2

1.- No, as Val said you don't need to create S3 buckets in several regions. "all buckets work globally" AWS S3 Bucket with Multiple Regions

2.- Yes you do. You need to create the snapshot repository in both of your clusters. One repository for create your snapshot to the S3 bucket in us-east-1 And other for your snaphost in us-west-2, in order to read from your destination cluster.

3.- Yes It is. Additionally, you need to sign your calls to AWS ES to be able to create the repo and to take the snapshot. The best option for me was to use the Python script described below. To restore it is not necessary.

Follow this instructions: https://medium.com/docsapp-product-and-technology/aws-elasticsearch-manual-snapshot-and-restore-on-aws-s3-7e9783cdaecb and https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html

Create a repository

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://localhost:9999/' # include https:// and trailing / Your elasticsearch endpoint, if you use VPC, you can create a tunnel
region = 'us-east-1' # e.g. us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

path = '_snapshot/yourreponame' # the Elasticsearch API endpoint
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "yourreponame_bucket",
    "region": "us-east-1",
    "role_arn": "arn:aws:iam::1111111111111:role/AmazonESSnapshotRole" <-- Don't forget to create the AmazonESSnapshotRole
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers, verify=False)

print(r.status_code)
print(r.text)

Create a snapshot

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://localhost:9999/' # include https:// and trailing /
region = 'us-east-1' # e.g. us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

path = '_snapshot/yourreponame/yoursnapshot_name' # the Elasticsearch API endpoint
url = host + path

payload = {
  "indices": "*",
  "include_global_state": "false",
  "ignore_unavailable": "false"
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers, verify=False)

print(r.status_code)
print(r.text)

Restore

Must be called without signing

curl -XPOST -k "https://localhost:9999/_snapshot/yourreponame/yoursnapshot_name/_restore" \
-H "Content-type: application/json" \
-d $'{
  "indices": "*",
  "ignore_unavailable": false,
  "include_global_state": false,
  "include_aliases": false
}'

It is highly recommended that the clusters have the same version.