How to handle secrets in Google App Engine?

Question

My application needs a bunch of secrets to run: database credentials, API credentials, etc. It's running in Google App Engine Standard Java 11. I need these secrets as environment variables or as arguments to my application, so that my framework can pick them up and establish the connections accordingly. My particular framework is Spring Boot, but I believe Django, Rails and many others use the same methods.

What's the best way of doing this?

One of the answers I get to this question is to use Google Cloud Key Management, which looks promising, but I can't figure out how to turn those values into environment variables in App Engine. Is it possible? I've read Setting Up Authentication for Server to Server Production Applications, but I don't see any indication there about how to turn the secrets into environment variables in App Engine (am I missing it?).

The other alternatives I've seen include hard-coding them in app.yaml or another file that is never committed and lives in my machine, which means I'm the only one who can deploy... I can't even deploy from another machine. This is problematic for me.

Another potential solution I've seen is to delegate the problem to Google Cloud Build, so that it fetches a value/file from CKM and pushes it to App Engine (1, 2). I'm not using GCB and I doubt I will, since it's so basic.

I really wish App Engine had a environment variables page like Heroku does.

Answer 1

[Update] (as of Feb 2020) GCP's Secret Manager is in beta, see:

https://cloud.google.com/secret-manager/docs/overview

For Java-specific implementation, see: https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-java

Your specific solution would depend how your app is set up, but you should be able to access the secret(s) and create environment variables with the values or otherwise pass them to your app.

You can use GCP IAM to create a service accounts to manage access or add a role like Secret Manager Secret Accessor to an existing member/service (e.g., in this case, I added that permision to the App Engine default service account).

I tried it out with Node.js on GAE standard, and it seems to work well; I didn't do any performance tests but it should be fine, particularly if you primarily need the secrets on app start or as part of a build process.

For local (non-GCP) development/testing, you can create a service account with appropriate secret manager permissions and get the json service key. You then set an environment variable named GOOGLE_APPLICATION_CREDENTIALS to the path of the file, e.g.:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/local_service_key.json

and the app running in that shell session should pick up the permissions without any additional auth code. See: https://cloud.google.com/docs/authentication/getting-started (You would want to exclude the key file from version control.)

Answer 2

At this date, App Engine Standard Standard does not have a Google provided solution for storing application secrets.

[UPDATE]

I noticed your comment on another answer that you require environment variables to be valid before you have application control. In that case, you have no options for App Engine today. I would deploy to a different service (Kubernetes) better suited for your system goals that can provided managed secrets.

[END UPDATE]

You have two choices for secrets for App Engine Standard:

1. Store the secrets as environment variables in app.yaml
2. Store the secrets someplace else.

For both options, you can add a layer of security by encrypting them. However, adding encryption adds another secret (decryption key) that you must somehow provide to your app. The chicken-or-egg situation.

App Engine Standard uses a Service Account. This service account can be used as an identity to control access to other resources. Examples of other resources are KMS and Cloud Storage. This means that you can securely access KMS or Cloud Storage without adding another secret to App Engine.

Let's assume that your company wants all application secrets encrypted. We can use the App Engine Service Account as the identity authorized to access KMS for a single key.

Note: The following examples use Windows syntax. Replace the line continuation ^ with \ for Linux/macOS.

Create the KMS Keyring. Keyrings cannot be deleted, so this is a one-time operation.

set GCP_KMS_KEYRING=app-keyring
set GCP_KMS_KEYNAME=app-keyname

gcloud kms keyrings create %GCP_KMS_KEYRING% --location global

Create the KMS Key.

gcloud kms keys create %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--purpose encryption

Add the service account to the KMS policy for the keyring and key that we created.

This will allow App Engine to decrypt data without requiring secrets for KMS. The service account identity provides access control. No roles are required for KMS. You will need to provide the KMS Keyring and Keyname which can be included in app.yaml.

set GCP_SA=<replace with the app engine service acccount email adddress>
set GCP_KMS_ROLE=roles/cloudkms.cryptoKeyDecrypter

gcloud kms keys add-iam-policy-binding %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--member serviceAccount:%GCP_SA% ^
--role %GCP_KMS_ROLE%

For this example, let's assume that you need to access a MySQL database. We will store the credentials in a JSON file and encrypt it. The file is named config.json.

{
        "DB_HOST": "127.0.0.1",
        "DB_PORT": "3306",
        "DB_USER": "Roberts",
        "DB_PASS": "Keep-This-Secret"
}

Encrypt config.json using Cloud KMS and store the encrypted results in config.enc:

call gcloud kms encrypt ^
--location=global ^
--keyring %GCP_KMS_KEYRING% ^
--key=%GCP_KMS_KEYNAME% ^
--plaintext-file=config.json ^
--ciphertext-file=config.enc

The encrypted file can be stored in Cloud Storage. Since it is encrypted, you could store the file with your build files, but I do not recommend that.

The final piece is to write the code in Java that is part of your program that uses KMS to decrypt the file config.enc using KMS. Google has a number of examples of KMS decryption:

Java KMS Decrypt

Java Samples

Answer 3

You can pass secrets as env variables at build time. This example retrieves a Stripe API key and updates app.yaml within Cloud Build, ensuring the local file is not accidentally checked in to source control

First make sure the CloudBuild service account has IAM role Secret Manager Secret Accessor

an app.dev.yaml file with a place holder for the env variable

runtime: python39
env: standard

instance_class: F4

automatic_scaling:
  max_instances: 1

env_variables:
  STRIPE_API_KEY: STRIPE_API_VAR

etc
etc

Cloudbuild.yaml to retrieve the secret and insert at build time

steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args:
      - -c
      - |
        echo 'my api key from secret manager is '$$STRIPE_API_VAR
        sed -i "s|STRIPE_API_VAR|$$STRIPE_API_VAR|g" app.dev.yaml
        cat app.dev.yaml # you can now see the secret value inserted as the env variable
        gcloud app deploy --appyaml=app.dev.yaml # deploy with the updated app.yaml, the local copy of the file is not changed
  secretEnv: ['STRIPE_API_VAR']

availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/stripe-api-key/versions/latest
    env: 'STRIPE_API_VAR'

Answer 4

Berglas looks interesting.

Another option is to put the secrets in app.yaml file(s) (you can have more than one) and encrypt it before committing it to version control.

There are many tools to encrypt secrets before putting them in version control, like https://github.com/StackExchange/blackbox

Pros:

• Very versatile
• I find it simple to understand compared to other options
• Easy to get started

Cons:

• You can't really remove access for a person (since the file could always be copied) so you have rotate secrets sometimes
• Can be hard to keep the unencrypted files out of the repo. Ones you get used to it, and have ignore files and/or scripts, it's usually OK.

Answer 5

For secret management, I'm personally fan of Berglas project. It's based on KMS and, in addition, manage DEK and KEK

It's today write in Go and it's not compliant with Java. I wrote a python library for some colleagues. I can write a Java package if you plan to use it. It's not very hard.

Let me know