46

In http://docs.docker.com/engine/reference/builder/#arg , It recommendeds secrets are not passed through ARGS.

Note: It is not recommended to use build-time variables for passing secrets like github keys, user credentials etc.

At what point are secrets passed through build-time variables in danger?

Roger Lam
  • 962
  • 2
  • 8
  • 17
  • 1
    When you type them into the terminal in plaintext? – OneCricketeer Nov 10 '15 at 01:18
  • Would that also apply to cat-ing a file to an environment variable and passing it through env variables? – Roger Lam Nov 10 '15 at 01:36
  • I think cat-ing a file is fine. Some people place their keys (such as AWS) in `.bashrc` so they are loaded in each bash session. I just think that note is referring to `docker build --build-arg param=...` – OneCricketeer Nov 10 '15 at 01:45
  • 2
    I'm trying to pass an ssh key at build time without leaving a layer history of the key. Adding, deleting, and squashing is an option but ideally I wouldn't have to do that for hiding secrets. – Roger Lam Nov 10 '15 at 01:49
  • 4
    Note: with docker 18.09+, you now have **`docker build --secret id=mysecret,src=/secret/file`**. See https://stackoverflow.com/a/51921954/6309 – VonC Aug 19 '18 at 21:25

4 Answers4

44

Update August 2018:

You now have docker build --secret id=mysecret,src=/secret/file.
See "safe way to use build-time argument in Docker".

Update January 2017:

Docker (swarm) 1.13 has docker secret.

However, as commented by Steve Hoffman (bacoboy):

[...]The secret command only helps swarm users is not a more general solution (like they did with attaching persistent volumes).
How you manage your secrets (what they are and who has access to them) is very system dependent and depends on which bits of paid and/or OSS you cobble together to make your "platform".
With Docker the company moving into providing a platform, I'm not surprised that their first implementation is swarm based just as Hashicorp is integrating Vault into Atlas -- it makes sense.

Really how the secrets are passed falls outside the space of docker run.
AWS does this kind of thing with roles and policies to grant/deny permissions plus an SDK.
Chef does it using encrypted databags and crypto "bootstrapping" to auth.
K8S has their own version of what just got released in 1.13.
I'm sure mesos will add a similar implementation in time.

These implementations seem to fall into 2 camps.

  • pass the secret via volume mount that the "platform" provides or (chef/docker secret/k8s
  • pass credentials to talk to an external service to get things at boot (iam/credstash/etc)

Original answer: Nov. 2015

This was introduced in commit 54240f8 (docker 1.9, Nov 2015), from PR 15182,

The build environment is prepended to the intermediate continer's command string for aiding cache lookups.
It also helps with build traceability. But this also makes the feature less secure from point of view of passing build time secrets.

issue 13490 reiterates:

Build-time environment variables: The build-time environment variables were not designed to handle secrets. By lack of other options, people are planning to use them for this. To prevent giving the impression that they are suitable for secrets, it's been decided to deliberately not encrypt those variables in the process.

As mentioned in 9176 comments:

env variables are the wrong way to pass secrets around. We shouldn't be trying to reinvent the wheel and provide a crippled security distribution mechanism right out of the box.

When you store your secret keys in the environment, you are prone to accidentally expose them -- exactly what we want to avoid:

  • Given that the environment is implicitly available to the process, it's incredibly hard, if not impossible, to track access and how the contents get exposed
  • It is incredibly common having applications grabbing the whole environment and print it out, since it can be useful for debugging, or even send it as part of an error report. So many secrets get leaked to PagerDuty that they have a well-greased internal process to scrub them from their infrastructure.
  • Environment variables are passed down to child processes, which allows unintended access and breaks the principle of least privilege. Imagine that as part of your application you call to third-party tool to perform some action, all of a sudden that third-party tool has access to your environment, and god knows what it will do with it.
  • It is very common for applications that crash to store the environment variables in log-files for later debugging. This means secrets in plain-text on disk.
  • Putting secrets in env variables quickly turns into tribal knowledge. New engineers don't know they are there, and are not aware they should be careful when handling environment variables (filtering them to sub-processes, etc).

Overall, secrets in env variables break the principle of least surprise, are a bad practice and will lead to the eventual leak of secrets.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • 10
    This answer is the best I've seen, but a lot of these comments refer to ENV and not ARG. Also, there's no mention of what the recommended way the recommended way to pass secrets to the build actually is... – Shane Dec 16 '15 at 02:48
  • Further reading of [issue 13490](https://github.com/docker/docker/issues/13490) shows that there actually isn't a recommended solution yet, but several (including using password vaults) are being explored. – Shane Dec 16 '15 at 03:23
  • 2
    I understand that the question was answered lovely by VonC [respect :] ], but there is still the question of how should we manage secrets and pass them to dockerfiles and/or compose eventually... – Kostas Demiris Feb 05 '16 at 11:47
  • @KostasDemiris I agree. Up to now, I only saw https://docs.docker.com/compose/compose-file/#environment. Other than that, various vault-based solution: https://github.com/docker/docker/issues/10310#issuecomment-107758040, https://github.com/hashicorp/vault/issues/165 – VonC Feb 05 '16 at 11:53
  • 1
    Where should you store secrets then? I thought environment variables were best practice, at least according to Heroku's 12 factor app principles: http://12factor.net/config – Richard Jun 15 '16 at 14:07
  • @Richard yes, EVs (Environment Variables) are explored right now, but the main drawback is the risk of them being leaked (through docker inspect for instance: https://github.com/docker/docker/issues/13490#issuecomment-213501080). Still the general idea is runtime decryption within the container and in-memory: https://github.com/docker/docker/issues/13490#issuecomment-185713425 – VonC Jun 15 '16 at 14:53
  • @VonC Hm, for me, your edit from 2017 somehow destroyed your answer, as it does not answer the original question anymore, instead it reflects another question asked by others from the comments...But it would perfectly fit here: [Docker and securing passwords](https://stackoverflow.com/questions/22651647/docker-and-securing-passwords/48460539#48460539) – Murmel Jan 26 '18 at 11:43
  • 1
    @Murmel I see, I must have considered the question "evolved" because of those comments. Anyway, I'll leave it here. – VonC Jan 26 '18 at 11:50
19

The simple reason is that the value of the secret is visible to anyone with the image by simply running history on it.

Take this sample docker file:

FROM alpine

ARG secret

RUN echo "${secret}"

(Nice and simple, just to illustrate how you might use a secret.)

then we build it $ docker build --build-arg secret=S3CR3T - < Dockerfile

Sending build context to Docker daemon 2.048 kB
Step 1 : FROM alpine
 ---> 13e1761bf172
Step 2 : ARG secret
 ---> Running in 695b7a931445
 ---> 5414c15a1cb6
Removing intermediate container 695b7a931445
Step 3 : RUN echo "${secret}"
 ---> Running in c90cf0d1414b
s3cr3t
 ---> f2bcff49ac09
Removing intermediate container c90cf0d1414b
Successfully built f2bcff49ac09

And an example of how to get the "secret" back out (look for |1 secret= on the first line):

$ docker history f2bcff49ac09
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
f2bcff49ac09        8 seconds ago       |1 secret=S3CR3T /bin/sh -c echo "${secret}"    0 B
5414c15a1cb6        8 seconds ago       /bin/sh -c #(nop) ARG secret                    0 B
13e1761bf172        6 months ago        /bin/sh -c #(nop) ADD file:614a9122187935fccf   4.797 MB

This is the case if you have built the image locally or pulled it from a registry.

If your goal is to keep the build-time secret out of the running container then using ARG does help you - consider this:

$ docker run --rm -ti f2bcff49ac09 sh
/ # env
HOSTNAME=7bc772fd0f56
SHLVL=1
HOME=/root
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
$ # Note no secret in the above output
moxim
  • 111
  • 1
  • 9
Ash Berlin-Taylor
  • 3,879
  • 29
  • 34
5

I think the new (17.05) docker feature Multi-stage Builds (https://docs.docker.com/engine/userguide/eng-image/multistage-build/) mitigates these (valid) concerns about simply using --build-arg.

FROM mybuildertools
ADD my-git-creds /root/.ssh
RUN git clone git@bitbucket.org:example/foo /src
FROM mybuildertools
COPY --from=0 /src /src
RUN ...build /src with no git credentials ending up in final image...

Unfortunately there doesn't seem to be an easy way to allow subsequent rebuilds (e.g. changes to the build step in the Dockerfile) unless you have the "my-git-creds" directory.

jamshid
  • 1,775
  • 17
  • 14
  • 5
    Is this correct? Even with a multi-stage build, the machine building the image would still end up with cached layers that include the secret, even if the application execution image doesn't have it. Seems like that would still be a concern depending on your build infrastructure... – rationull May 04 '18 at 20:28
  • True, it's only safe when you're pushing the final image to a docker registry. – jamshid May 09 '18 at 05:39
3

I wrote https://github.com/abourget/secrets-bridge to address the build-time secrets problem.

It creates a throw-away configuration that you can pass as a build argument, during the build process, it will connect to the host and fetch the secrets, use them, and then you can kill the host bridge. Even if the build-args are saved somewhere, they become useless the moment the server exits.

The server supports SSH Agent Forwarding, tunnelled through a TLS websocket communication. It works on Windows too !

Hope this helps.

abourget
  • 2,351
  • 19
  • 17