Configuring Concourse CI to use AWS Secrets Manager

Question

I have been trying to figure out how to configure the docker version of Concourse (https://github.com/concourse/concourse-docker) to use the AWS Secrets Manager and I added the following environment variables into the docker-compose file but from the logs it doesn't look like it ever reaches out to AWS to fetch the creds. Am I missing something or should this automatically happen when adding these environment variables under environment in the docker-compose file? Here are the docs I have been looking at https://concourse-ci.org/aws-asm-credential-manager.html

version: '3'

services:
  concourse-db:
    image: postgres
    environment:
      POSTGRES_DB: concourse
      POSTGRES_PASSWORD: concourse_pass
      POSTGRES_USER: concourse_user
      PGDATA: /database

  concourse:
    image: concourse/concourse
    command: quickstart
    privileged: true
    depends_on: [concourse-db]
    ports: ["9090:8080"]
    environment:
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse_user
      CONCOURSE_POSTGRES_PASSWORD: concourse_pass
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_EXTERNAL_URL: http://XXX.XXX.XXX.XXX:9090
      CONCOURSE_ADD_LOCAL_USER: test: test
      CONCOURSE_MAIN_TEAM_LOCAL_USER: test
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
      CONCOURSE_AWS_SECRETSMANAGER_REGION: us-east-1
      CONCOURSE_AWS_SECRETSMANAGER_ACCESS_KEY: <XXXX>
      CONCOURSE_AWS_SECRETSMANAGER_SECRET_KEY: <XXXX>
      CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
      CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}

pipeline.yml example:

jobs:
  - name: build-ui
    plan:
      - get: web-ui
        trigger: true
      - get: resource-ui
      - task: build-task
        file: web-ui/ci/build/task.yml
      - put: resource-ui
        params:
          repository: updated-ui
          force: true
      - task: e2e-task
        file: web-ui/ci/e2e/task.yml
        params:
          UI_USERNAME: ((ui-username))
          UI_PASSWORD: ((ui-password))

resources:
  - name: cf
    type: cf-cli-resource
    source:
      api: https://api.run.pivotal.io
      username: ((cf-username))
      password: ((cf-password))
      org: Blah

  - name: web-ui
    type: git
    source:
      uri: git@github.com:blah/blah.git
      branch: master
      private_key: ((git-private-key))

Adding these in the compose for `web` service should be sufficient. Can you explain how you are trying to access the secrets from SM? Also please post the compose file containing these environment variables. — franklinsijo, Apr 15 '20 at 01:31
@franklinsijo I have updated it with the compose file. Let me know your thoughts. — exosyphon, Apr 15 '20 at 23:40
Also can you explain how you are trying to access the secrets in your secrets manager, let's say from your pipeline tasks? — franklinsijo, Apr 16 '20 at 00:44
@franklinsijo Added example of how I am accessing the secrets in the pipeline.yml. Up to now we have used a pipeline-secrets.yml file to inject secrets but are now looking at moving everything to AWS Secrets Manager. — exosyphon, Apr 16 '20 at 02:14
I assume that the parameters like password and keys are the ones that are being passed on from `pipeline-secrets.yml` and from now has to be fetched from AWS Secrets Manager. Am I correct in that understanding? — franklinsijo, Apr 16 '20 at 02:35
Did you make any changes to the parameter names, now that it has to be fetched from Secrets Manager instead of your vars file? The syntax to access parameters from Secrets Manager is different. — franklinsijo, Apr 16 '20 at 03:06
I didn't, can you give an example of what the different syntax should be? — exosyphon, Apr 16 '20 at 03:24

franklinsijo · Accepted Answer · 2020-04-16T04:29:59.953

3

When storing parameters for concourse pipelines in AWS Secrets Manager, it must follow this syntax,

/concourse/TEAM_NAME/PIPELINE_NAME/PARAMETER_NAME`

If you have common parameters that are used across the team in multiple pipelines, use this syntax to avoid creating redundant parameters in secrets manager

/concourse/TEAM_NAME/PARAMETER_NAME

The highest level that is supported is concourse team level.

Global parameters are not possible. Thus these variables in your compose environment will not be supported.

CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}

Unless you want to change the prefix /concourse, these parameters shall be left to their defaults.

And, when retrieving these parameters in the pipeline, no changes required in the template. Just pass the PARAMETER_NAME, concourse will handle the lookup in secrets manager as per the team and pipeline name.

...
        params:
          UI_USERNAME: ((ui-username))
          UI_PASSWORD: ((ui-password))
...

edited Apr 16 '20 at 04:29

answered Apr 16 '20 at 03:44

franklinsijo

17,784
4
45
63

So if I changed the prefix to be /bob then specifying the template as CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /bob/{{.Secret}} would work? – exosyphon Apr 16 '20 at 12:08
It is a team secret, so it will be still looking for `/bob/{{.Team}}/{{.Secret}}`. You can only modify the base path. – franklinsijo Apr 16 '20 at 12:13
Gotcha, thank you! I updated per your suggestions and was able to see a AccessDenied message from AWS so it is connecting correctly now. Unfortunately after updating the permissions I now get `undefined vars: cf-username, cf-password`. I should be able to pull those creds for a resource right? – exosyphon Apr 16 '20 at 12:57
Yes. Confirm whether you have them in correct format and path! – franklinsijo Apr 16 '20 at 12:59
So for those I have the secret name as concourse/team and then the secret is cf-username. Should that be the correct way? – exosyphon Apr 16 '20 at 13:06
Yes, it must be stored as `/concourse//cf-username` in the secret manager. – franklinsijo Apr 16 '20 at 13:08
Here was the error message I was getting before giving permissions. Finding variable 'cf-username': AccessDeniedException: User: arn:aws:iam::XXXXXX:user/deploy is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-east-1:XXXXX:secret:/concourse/team/cf-username-1WsM2A status code: 400, request id: 3d34170b-78f7-4847-bd51-208a5a1644cb – exosyphon Apr 16 '20 at 13:08
And Did you update IAM policy of the user to allow `secretsmanager:GetSecretValue` and why is the parameter name `cf-username-1WsM2A ` is stored like this? – franklinsijo Apr 16 '20 at 13:09
Yes, after giving the user secretsmanager:GetSecretValue then I got that undefined vars error. I am not sure why it has that ending, it looks like it changes on each request – exosyphon Apr 16 '20 at 13:10
I see the parameter name is `cf-username-1WsM2A` instead of `cf-username` and also you have literal `team` instead of the name of the team under which the pipeline belongs. – franklinsijo Apr 16 '20 at 13:12
The team I have replaced in our example but it is correct in the setup. In AWS the secret name is cf-username but in that error it looks like it appends some random characters. Those characters change each time. Here is another example cf-username-uO9KM3 – exosyphon Apr 16 '20 at 13:15
Those random characters are part of the ARN, not the secret name. It will not pose any issue for concourse to access. – franklinsijo Apr 16 '20 at 13:50
1

Thank you again, we determined that having the / on the front was critical and make sure to have a separate secret for each entry that only has a string value and not using the key/value pair syntax. – exosyphon Apr 16 '20 at 14:59

Configuring Concourse CI to use AWS Secrets Manager

1 Answers1