What are some concrete use-cases for Consul's Key-Value store?

Question

We're considering using Consul's key-value store to enhance our configuration management tool (Ansible, at the moment). We're hoping to use it to solve two related problems:

1. Preventing scatter: Some items (namely: passwords, certificates etc) are scattered across our configuration files. Updating them requires manual search-and-replace which can be tiresome.
2. Ease of update: rather then edit-and-commit configuration changes into git, we could use Consul to store those items that change often.

We're looking for a set of recommendations on how to use/integrate Consul (or similar tools) for dynamic configurations. Naturally, there is no one answer, but a set of useful practices here. I'll provide a few approaches in my answer, but I'd like to hear additional ideas.

Answer 1

To mitigate #1 I'd suggest looking into Hashicorps Vault, https://www.vaultproject.io/, which is a tool to handle secrets, which can use Consul as a backend.

Answer 2

We've been tinkering with Consul as a key-value store for a while but I think the most interesting use comes with Consul Template and using that to update configuration on the fly.

I think the end state we're probably moving towards is going to be to use Ansible to configure a base image of things we know are slow changing plus configure Consul Template, then AMI this (these first 2 steps probably done via Packer) and then deploy into auto scaling groups in AWS using Terraform (which we already use for provisioning).

Then we will use Consul's key-value store to change properties that Consul Template will then propagate across a cluster of instances. We also intend to have instances register themselves in Consul which will also affect configuration on other instances such as load balancing members on Apache/NGINX configurations or lists of unicast addressable members for clustering.

On a slightly related note, and as mentioned by mahnve, Vault is a pretty nice add on to Consul for storing secrets. We're already using it for pretty static secrets but intend to start using some of the dynamic secret generation which allows you to request short lived API or SSH keys which can be tracked and revoked.

Answer 3

1. We've yet to do this, but are thinking about integrating consul into our Ansible plays. Ansible recently added a lookup option from consul:

https://github.com/ansible/ansible/blob/devel/test/integration/roles/test_consul_kv/tasks/main.yml#L70

- debug: msg='key contains {{item}}'
  with_consul_kv:
  - 'key/to/retrieve'

So we could directly populate our plays with values from Consul.

2. Another approach we're considering is to utilize consul's templating tool - and template entire configuration files after ansible plants them on our hosts.