how do you manage secret values with docker-compose v3.1?

Question

Version 3.1 of the docker-compose.yml specification introduces support for secrets.

I tried this:

version: '3.1'

services:
  a: 
    image: tutum/hello-world
  secret: 
    password: the_password
  b:
    image: tutum/hello-world

$ docker-compose up returns:

Unsupported config option for services.secret: 'password'

How can we use the secrets feature in practice?

Answer 1

You can read the corresponding section from the official documentation.

To use secrets you need to add two things into your docker-compose.yml file. First, a top-level secrets: block that defines all of the secrets. Then, another secrets: block under each service that specifies which secrets the service should receive.

As an example, create the two types of secrets that Docker will understand: external secrets and file secrets.

1. Create an 'external' secret using docker secret create

First thing: to use secrets with Docker, the node you are on must be part of a swarm.

$ docker swarm init

Next, create an 'external' secret:

$ echo "This is an external secret" | docker secret create my_external_secret -

(Make sure to include the final dash, -. It's easy to miss.)

2. Write another secret into a file

$ echo "This is a file secret." > my_file_secret.txt

3. Create a docker-compose.yml file that uses both secrets

Now that both types of secrets are created, here is the docker-compose.yml file that will read both of those and write them to the web service:

version: '3.1'

services:
  web:
    image: nginxdemos/hello
    secrets:                    # secrets block only for 'web' service
     - my_external_secret
     - my_file_secret

secrets:                        # top level secrets block
  my_external_secret:
    external: true
  my_file_secret:
    file: my_file_secret.txt

Docker can read secrets either from its own database (e.g. secrets made with docker secret create) or from a file. The above shows both examples.

4. Deploy your test stack

Deploy the stack using:

$ docker stack deploy --compose-file=docker-compose.yml secret_test

This will create one instance of the web service, named secret_test_web.

5. Verify that the container created by the service has both secrets

Use docker exec -ti [container] /bin/sh to verify that the secrets exist.

(Note: in the below docker exec command, the m2jgac... portion will be different on your machine. Run docker ps to find your container name.)

$ docker exec -ti secret_test_web.1.m2jgacogzsiaqhgq1z0yrwekd /bin/sh

# Now inside secret_test_web; secrets are contained in /run/secrets/
root@secret_test_web:~$ cd /run/secrets/

root@secret_test_web:/run/secrets$ ls
my_external_secret  my_file_secret

root@secret_test_web:/run/secrets$ cat my_external_secret
This is an external secret

root@secret_test_web:/run/secrets$ cat my_file_secret
This is a file secret.

If all is well, the two secrets we created in steps 1 and 2 should be inside the web container that was created when we deployed our stack.

Answer 2

Given you have a service myapp and a secrets file secrets.yml:

Create a compose file:

version: '3.1'

services:
  myapp:
    build: .
    secrets:
      secrets_yaml

Provision a secret using this command:

docker secret create secrets_yaml secrets.yml

Deploy your service using this command:

docker deploy --compose-file docker-compose.yml myappstack

Now your app can access the secret file at /run/secrets/secrets_yaml. You can either hardcode this path in your application or create a symbolic link.

---

The different question

This answer is probably to the question "how do you provision your secrets to your docker swarm cluster".

The original question "how do you manage secret values with docker compose" implies that the docker-compose file contains secret values. It doesn't.

There's a different question: "Where do you store the canonical source of the secrets.yml file". This is up to you. You can store it in your head, print on a sheet of paper, use a password manager, use a dedicated secrets application/database. Heck, you can even use a git repository if it's safely secured itself. Of course, never store it inside the system you're securing with it :)

I would recommend vault. To store a secret:

# create a temporary secret file
cat secrets.yml | vault write secret/myappsecrets -

To retrieve a secret and put it into your docker swarm:

vault read -field=value secret/myappsecrets | docker secret create secrets_yaml -

Of course, you can use docker cluster itself as a single source of truth for you secrets, but if your docker cluster breaks, you'd lost your secrets. So make sure to have a backup elsewhere.

---

The question nobody asked

The third question (that nobody asked) is how to provision secrets to developers' machines. It might be needed when there's an external service which is impossible to mock locally or a large database which is impossible to copy.

Again, docker has nothing to do with it (yet). It doesn't have access control lists which specify which developers have access to which secrets. Nor does it have any authentication mechanism.

The ideal solution appears to be this:

• A developer opens some web application.
• Authenticates using some single sign on mechanism.
• Copies some long list of docker secret create commands and executes them in the terminal.

We have yet to see if such an application pops up.

Answer 3

You can also specify secrets stored locally in a file using file: key in secrets object. Then you don't have to docker secret create them yourself, Compose / docker stack deploy will do it for you.

version: '3.1'

secrets:
  password:
    file: ./password

services:
  password_consumer:
    image: alpine
    secrets:
      - password

Reference: Compose file version 3 reference: Secrets

Answer 4

One question was raised here in the comments, why should I initialize a swarm if I only need secrets? And my answer is that secrets is created for the swarm, where you have more than one node and you want to manage and share secrets in a secure way. But if you have one node, this will not (almost) add any extra security if someone can access your host machine where you have the one node swarm, as secrets can be retrieved from the running containers, or directly on the host if the secret is created from a file, like a private key.

Check this blog: https://www.docker.com/blog/docker-secrets-management/

And read the comments: "Thank you very much for the introductory article. The steps are mentioned to view the contents of secrets in container will not work when the redis container is created on a worker node."

Answer 5

Is that the exact indentation of your docker-compose.yml file? I think secret secrets should be nested under a (i.e. one of the services), not directly under services section.

Answer 6

For local machine I ended up having 2 .env files: .env, secrets.env (sensitive envs, this file is in .gitignore). Starting from compose version 2.17.0-rc.1 you can specify multiple env files and use it like sudo docker compose --env-file .env --env-file secrets.env up

Pros:

1. You can store all secrets in one file instead of creating multiple secrets.
2. You don’t need swarm.

Cons:

1. More characters in command.
2. When you invoke commands like sudo docker compose logs you will need to specify --env-file .env --env-file secrets.env. Otherwise you will see errors like WARN[0000] The "VAR_NAME" variable is not set. Defaulting to a blank string.

Answer 7

I guess the keyword is secrets not secret. That is at least what I understand from reading the schema.

Answer 8

The keyword is secrets instead of secret. It should also properly indented under service a.