5

How can I automate version control of Kibana Saved Objects in an AWS environment?

I have an Elasticsearch domain hosted on AWS and I have configured Kibana to visualize the data. Now, I would like to version control the Kibana configuration for several reasons:

  • To have a consistent configuration across different environments (dev, stage, prod)
  • Disaster recovery in case the Elasticsearch domain is deleted
  • To be able to upgrade the Elasticsearch cluster to the latest version (I cannot use the AWS console to upgrade Elasticsearch as suggested in the AWS developer guide, since the Elasticsearch domain is part of a CloudFormation stack. Moreover, according to the CloudFormation user Guide, updating the ElasticsearchVersion requires replacement, hence the need for version control).

The only solution that I have found so far includes several manual steps:

Backup:

  1. Logon to your first Kibana environment
  2. Select the Saved Objects tab in the Kibana management console
  3. Press the Export Everything button
  4. Add the downloaded .json file to version control

Restore:

  1. Logon to the second Kibana environment
  2. Select Index Patterns tab in the Kibana management console
  3. Re-create the index patterns that are required for the visualizations (see the last warning in the Kibana user guide)
  4. Select the Saved Objects tab in the Kibana management console
  5. Press the Import button and upload the .json file that was version controlled
  6. Re-map each index pattern (since the ones stored in the .json file originates from the first Kibana environment does not exist in the second environment)
  7. Done(?)

How can this be automated?

Current environment:

  • Elasticsearch version 6.2.3
  • Kibana version 6.2.3
matsev
  • 32,104
  • 16
  • 121
  • 156
  • why not simply GETting the `.kibana` index entirely? – Val Aug 16 '18 at 11:49
  • 1
    @Val Thanks for the suggestion. I checked the possibility, but it does not contain neither visualization nor dashboard config. Additionally, the [Kibana 6.3 documentation](https://www.elastic.co/guide/en/kibana/6.3/saved-objects-api.html) states that `Do not do this! Writing directly to the .kibana index is not safe and it will result in corrupted data that permanently breaks Kibana in a future version.` – matsev Aug 16 '18 at 12:06
  • The `.kibana` index contains all index patterns, all visualizations and all dashboards, everything is in there. I'm not suggesting to write to the `.kibana` index, just GET it and save it for the cases you mentioned. – Val Aug 16 '18 at 12:14
  • @Val: I assume that you just `GET /.kibana`? I do not receive the config for neither visualizations nor dashboards in the response. Perhaps it is different for different Kibana versions (hence the remark about not using the `.kibana` index in the documentation)? – matsev Aug 16 '18 at 13:20
  • Actually, by GETting the `.kibana` index, I meant `GET .kibana/_search?size=10000` (add size to make sure you get everything). Sorry for the confusion – Val Aug 16 '18 at 13:25
  • @Val: No worries. That made a difference, now I get config data for index patterns, visualizations, etc. It seems like this approach can be used for backup, but given the remark mentioned earlier I will have play with it before I am convinced about restore (especially when upgrading between Kibana versions). – matsev Aug 16 '18 at 13:56
  • Usually, the Kibana documentation tells you how to migrate between versions if there are any incompatible changes. See here for example: https://www.elastic.co/guide/en/kibana/current/migrating-6.0-index.html – Val Aug 16 '18 at 14:14
  • 1
    Actually instead of getting the .kibana index, it might be a better idea to snapshot the .kibana index? https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html – willemdh Aug 28 '18 at 17:15

0 Answers0