Is it possible to deploy a Spring Boot app to App Engine and connect to the database?

Question

I feel that I'm going around in circles here, so, please, bear with me. I want to deploy my Spring Boot application to App Engine but unlike the simple sample Google provides, mine, requires a database and that means credentials. I'm running Java 11 on Standard on Google App Engine.

I managed to make my app successfully connect by having this in the application.properties:

spring.datasource.url=jdbc:postgresql://google/recruiters_wtf?cloudSqlInstance=recruiters-wtf:europe-west2:recruiters-wtf&socketFactory=com.google.cloud.sql.postgres.SocketFactory&user=the_user&password=monkey123

The problem is that I don't want to commit any credentials to the repository, so, this is not acceptable. I could use an environment variable, but then I'll have to define them in the app.yaml file. I either keep a non-committed app.yaml file that is needed to deploy, which is cumbersome, or I commit it and I'm back at square one, committing credentials to the repository.

Since apparently Google App Engine cannot have environment variables defined in any other way (unlike Heroku), does this mean it's impossible to deploy a Spring Boot app to App Engine and have it connect to the database without using some unsafe/cumbersome practices? I feel I'm missing something here.

Answer 1

Based on my understanding of what you have described, you would like to essentially connect your Spring boot application running on Google App Engine to a database without exposing the sensitive information. If that is the case, I was able to find out that Cloud KMS offers users the ability for secret management. Specifically, applications which require small pieces of sensitive data at build or runtime are referred to as secrets. These secrets can be encrypted and decrypted with a symmetric key. In your case you can store the database credentials as secrets. You may find further details of the process for encrypting / decrypting a secret here.

They are currently three ways to manage secrets:

1. Storing secrets in code, encrypted with a key from Cloud KMS. This solution is implementing secrets at the application layer.
2. Storing secrets in a storage bucket in Cloud Storage, encrypted at rest. You can use Cloud Storage: Bucket to store your database credentials and can also grant that bucket a specific Service Account. This solution allows for separation of systems. In the case that the code repository is breached, your secrets would themselves may still be protected.
3. Using third-party secret management system.

In terms of storing secrets themselves, I found the follow steps outlined here quite useful for this. This guide walks users through setting up and storing secrets within a Cloud Storage bucket. The secret is encrypted at the application layer with an encryption key from Cloud KMS. Given your use case, this would be a great option as your secret would be stored within a bucket instead of your app.yaml file. Also, the secret being stored in a bucket would grant you the ability to restrict access to it with service account roles.

Essentially, your app will need to perform an API call to Google Cloud Storage in order to download the KMS encrypted file that contains the secret. It would then use the KMS generated key to decrypt the file so that it would be able to read out the password and use it to make a manual connection to the database. Adding these extra steps would be implementing more security layers, which is the entire idea noted in “'Note: Saving credentials in environment variables is convenient, but not secure - consider a more secure solution such as Cloud KMS to help keep secrets safe.'” in the Google example repository for Cloud SQL.

I hope this helps!

Answer 2

Assuming you are OK to use KMS or GCS to get the credentials, you can programatically set them in Spring Boot. See this post

Configure DataSource programmatically in Spring Boot

Answer 3

As you pointed out there's no built-in way to set environment variables in App Engine other than with the app.yaml file. I'm not an expert in Spring Boot but unless you can set/override some hook to initialize env var from Java code prior to application.properties evaluation, you'll need to set these at build time.

Option 1: Using Cloud Build

I know you're not really keen to use Cloud Build but that would be something like this.

First, following the instructions here, (after creating KeyRing and CryptoKey in KMS and grant access to Cloud Build service account) from you terminal encrypt your environment variable using KMS and get back its base64 representation:

echo -n $DB_PASSWORD | gcloud kms encrypt \
  --plaintext-file=- \  # - reads from stdin
  --ciphertext-file=- \  # - writes to stdout
  --location=global \
  --keyring=[KEYRING-NAME] \
  --key=[KEY-NAME] | base64

Next, let's say you have an app.yaml file like this:

runtime: java11
instance_class: F1

env_variables:
  USER:  db_user
  PASSWORD: db_passwd

create a cloudbuild.yaml file to define your build steps:

steps:
# replace env vars in app.yaml by their values from KMS
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args: ['-c', 'sed -i "s/TEST/$$PASSWORD/g" src/main/appengine/app.yaml']
  secretEnv: ['PASSWORD']
- name: 'gcr.io/cloud-builders/mvn'
  args: ['clean']
- name: 'gcr.io/cloud-builders/mvn'
  args: ['package']
- name: 'gcr.io/cloud-builders/mvn'
  args: ['appengine:deploy']
  timeout: '1600s'
secrets:
- kmsKeyName: projects/<PROJECT-ID>/locations/global/keyRings/<KEYRING_NAME>/cryptoKeys/<KEY_NAME>
  secretEnv:
    PASSWORD: <base64-encoded encrypted password>
timeout: '1600s'

You ould then deploy your app by running the following command:

gcloud builds submit .

The advantage of this method is that your local app.yaml file only contains placehoder values and can be safely committed. Or you can even set this build to trigger automatically every time you commit to a remote repository.

Option 2: Locally with a bash script

Instead of running mvn appengine:deploy to deploy your app, you could create a bash script that would replace the values in app.yaml, deploy the app, and remove the values straight away.. Something like:

#!/bin/bash
sed -i "s/db_passwd/$PASSWORD/g" src/main/appengine/app.yaml'
mvn appengine:deploy
sed -i "s/$PASSWORD/db_passwd/g" src/main/appengine/app.yaml'

and execute that bash script instead of running the maven command.

Answer 4

I would probably suggest the combination of Spring Cloud Config & Google Runtime Configuration API with your Spring Boot App.

Spring Cloud Config is component which is responsible for retrieving the configuration from remote locations and serving that configuration to your Spring Boot during initialization/boot up. The remote locations can be anything. e.g, GIT repository is widely used but for your use case, you can store the configuration in Google Runtime Configuration API.

So a sample flow will be like this.

Your Spring Boot App(with Config Client) --> Spring Cloud Config Server --> Google Runtime Configuration API

This requires for you to bring up a Spring Cloud Config Server as another App in GCP and have communications enabled from many of your Spring Boot Apps to a centralized Config Server which interacts with Google Runtime API.

Some documentation links.

https://cloud.spring.io/spring-cloud-config/reference/html/

https://docs.spring.io/spring-cloud-gcp/docs/1.1.0.M1/reference/html/_spring_cloud_config.html

https://cloud.google.com/deployment-manager/runtime-configurator/reference/rest/

Sample Spring Cloud GCP Config Example.

https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-config-sample

Answer 5

You can try to encrypt the password in the application properties. Take a look at http://mbcoder.com/spring-boot-how-to-encrypt-properties-in-application-properties/

Answer 6

You should use GCP's Key Management Service: https://cloud.google.com/kms/

Answer 7

We use a few options:

1 - Without Docker

Can you use this approach via the env or console?

We use (Spring Boot defined) environment variables. This is the default way of doing this:

SPRING_DATASOURCE_USERNAME=myusername
SPRING_DATASOURCE_PASSWORD=mypassword

According to the Spring Boot spec this will overrule any value of application.properties variables. So you can specify default username and passwords for development and have this overruled at (test or) production deploy time.

Another way is documented in this post:

spring.datasource.url = ${OPENSHIFT_MYSQL_DB_HOST}:${OPENSHIFT_MYSQL_DB_PORT}/"nameofDB"
spring.datasource.username = ${OPENSHIFT_MYSQL_DB_USERNAME}
spring.datasource.password = ${OPENSHIFT_MYSQL_DB_PASSWORD}

2 - A Docker like approach via a console

Your question is described in this post. The default solution is working with 'secrets'. They are specifically made for this. You can convert any secret (as a file) to an environment variable in your process of building and deploying your application. This is a simple action that is described in many posts. Look for newer approaches.

Answer 8

I feel like it is easy to do. I used below properties in my spring boot app which connect to google mysql instance which is public one.

spring.datasource.url=jdbc:postgresql://IP:5432/yourdatabasename
spring.datasource.platform = postgres
spring.datasource.username = youruser
spring.datasource.password = yourpass

Rest will be taken care by the JpaRepository. I hope this would help you.

Let me know if any help is needed.