DB connection during integration tests works on local, but not working on Google Cloud Build

Question

I'm deploying Java 11 REST API to GKE using GitHub, Gradle, and Docker.

The following errors are only happened on Google Cloud Build, not on the local environment. According to the error, it seems the app can't find the DB server(Google Cloud SQL) from Google Cloud Build. I tried both public and private IP, but the results were the same:

...
Step #0 - "Build":     2021-03-11 04:12:04.644  INFO 115 --- [    Test worker] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
Step #0 - "Build":     2021-03-11 04:12:35.855 ERROR 115 --- [    Test worker] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Exception during pool initialization.
Step #0 - "Build": 
Step #0 - "Build":     com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
Step #0 - "Build": 
Step #0 - "Build":     The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
...
Step #0 - "Build":     Caused by: java.net.SocketTimeoutException: connect timed out
...

This happened after I added integration tests. The app deployed successfully after I removed the tests. So, I can remove the integration tests to avoid this issue. The thing is, I want to keep the tests if possible because there are things that we can't test with unit tests.

This is the Dockerfile I'm using for deployments to GKE. RUN gradle build --no-daemon -i --stacktrace is where the error occurs during the test task:

ARG APP_NAME=test-api
ARG GRADLE_USER_HOME_PATH=/home/gradle/cache_home/

#cache dependencies to reduce downloads
FROM gradle:6.8-jdk11 AS cache
ARG APP_NAME
ARG GRADLE_USER_HOME_PATH
WORKDIR /${APP_NAME}/
RUN mkdir -p ${GRADLE_USER_HOME_PATH}
ENV GRADLE_USER_HOME ${GRADLE_USER_HOME_PATH}
COPY --chown=gradle:gradle build.gradle /${APP_NAME}/
RUN gradle clean build --no-daemon -i --stacktrace -x bootJar

#build
FROM gradle:6.8-jdk11 AS build
ARG APP_NAME
ARG GRADLE_USER_HOME_PATH
WORKDIR /${APP_NAME}/
#Copies cached dependencies
COPY --from=cache ${GRADLE_USER_HOME_PATH} /home/gradle/.gradle/
#Copies the Java source code inside the container
COPY --chown=gradle:gradle . /${APP_NAME}/
#Compiles the code and runs unit tests (with Gradle build)
RUN gradle build --no-daemon -i --stacktrace

#Discards the Gradle image with all the compiled classes/unit test results etc.
#Starts again from the JRE image and copies only the JAR file created before
FROM openjdk:11-jre-slim
ARG APP_NAME
COPY --from=build /${APP_NAME}/build/libs/${APP_NAME}.jar /${APP_NAME}/${APP_NAME}.jar
ENTRYPOINT ["java","-jar","/test-api/test-api.jar"]

How to implement integration tests that using DB to GKE? Or maybe I need to change my approach?

Which DB are you using for integration test ? Is it inbuilt DB ? — arpitj938, Mar 11 '21 at 06:14
@arpitj938 I'm using Google Cloud SQL(MySQL), same as the production. — user2652379, Mar 11 '21 at 06:25
I would suggest few points to you 1) use inbuilt DB similar to sqlite for testing 2) This testing must be done before creating docker container. — arpitj938, Mar 11 '21 at 08:03
As part of solution, I think you might be missing port mapping in docker run command. Can you try port mapping and check is it working ? — arpitj938, Mar 11 '21 at 08:06
@arpitj938 The deployed container's functions that use DB are functional, but I'll try the port mapping just in case. The second option you mentioned seems promising. If I'm not so pushy, would you like to expand it? Sorry to ask, I'm somewhat new to these CI/CD things. — user2652379, Mar 11 '21 at 08:21
Did port mapping or expose of port worked for you ? Explaining CI/CD in comment is kind of tricky and answering about in this question is out of scope, if you can modify the question or create a new question I will post it there my views as answers. — arpitj938, Mar 12 '21 at 09:07
@arpitj938 I somehow managed to find a workaround. While not preferred, I added "0.0.0.0/0" into the "authorized network" of public IP of Cloud SQL then the tests are passed. I hope there is a better way. — user2652379, Mar 12 '21 at 09:23
@arpitj938 About mapping or exposing of port... I don't know how to do it, so I couldn't try it. — user2652379, Mar 12 '21 at 09:29
For exposing you can add one line in dockerFile EXPOSE This means you are allowing external world to connect container with this port https://www.whitesourcesoftware.com/free-developer-tools/blog/docker-expose-port/ — arpitj938, Mar 12 '21 at 09:56

user2652379 · Accepted Answer · 2021-05-07T05:27:58.113

I managed to solve the problem referencing this Q&A: Run node.js database migrations on Google Cloud SQL during Google Cloud Build

I had to add 2 steps(Cloud SQL Proxy and Test) on cloudbuild.yaml to use Cloud SQL Proxy. The other steps were auto-generated by GKE:

steps:
  - name: gradle:6.8.3-jdk11
    entrypoint: sh
    args:
      - '-c'
      - |-
        apt-get update && apt-get install -y wget \
        && wget "https://storage.googleapis.com/cloudsql-proxy/v1.21.0/cloud_sql_proxy.linux.amd64" -O cloud_sql_proxy \
        && chmod +x cloud_sql_proxy \
        || exit 1
    id: Cloud SQL Proxy
  - name: gradle:6.8.3-jdk11
    entrypoint: sh
    args:
      - '-c'
      - |-
        (./cloud_sql_proxy -instances=<CONNECTION_NAME>=tcp:<PORT> & sleep 2) \
        && gradle test --no-daemon -i --stacktrace \
        || exit 1
    id: Test
  - name: gcr.io/cloud-builders/docker
    args:
      - build
      - '-t'
      - '$_IMAGE_NAME:$COMMIT_SHA'
      - .
      - '-f'
      - $_DOCKERFILE_NAME
    dir: $_DOCKERFILE_DIR
    id: Build
  - name: gcr.io/cloud-builders/docker
    args:
      - push
      - '$_IMAGE_NAME:$COMMIT_SHA'
    id: Push
  - name: gcr.io/cloud-builders/gke-deploy
    args:
      - prepare
      - '--filename=$_K8S_YAML_PATH'
      - '--image=$_IMAGE_NAME:$COMMIT_SHA'
      - '--app=$_K8S_APP_NAME'
      - '--version=$COMMIT_SHA'
      - '--namespace=$_K8S_NAMESPACE'
      - '--label=$_K8S_LABELS'
      - '--annotation=$_K8S_ANNOTATIONS,gcb-build-id=$BUILD_ID'
      - '--create-application-cr'
      - >-
        --links="Build
        details=https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=$PROJECT_ID"
      - '--output=output'
    id: Prepare deploy
  - name: gcr.io/cloud-builders/gsutil
    args:
      - '-c'
      - |-
        if [ "$_OUTPUT_BUCKET_PATH" != "" ]
        then
          gsutil cp -r output/suggested gs://$_OUTPUT_BUCKET_PATH/config/$_K8S_APP_NAME/$BUILD_ID/suggested
          gsutil cp -r output/expanded gs://$_OUTPUT_BUCKET_PATH/config/$_K8S_APP_NAME/$BUILD_ID/expanded
        fi
    id: Save configs
    entrypoint: sh
  - name: gcr.io/cloud-builders/gke-deploy
    args:
      - apply
      - '--filename=output/expanded'
      - '--cluster=$_GKE_CLUSTER'
      - '--location=$_GKE_LOCATION'
      - '--namespace=$_K8S_NAMESPACE'
    id: Apply deploy

...

And Dockerfile:

ARG APP_NAME=test-api
ARG APP_HOME=/test-api

FROM openjdk:11-jdk-slim AS build
USER root
ARG APP_HOME
WORKDIR ${APP_HOME}/
COPY . .
# test is performed from Test step from cloudbuild.yaml
RUN ./gradlew build --no-daemon -i --stacktrace -x test

FROM openjdk:11-jdk-slim
ARG APP_NAME
ARG APP_HOME
WORKDIR ${APP_HOME}/
COPY --from=build ${APP_HOME}/build/libs/${APP_NAME}.jar ./${APP_NAME}.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","/test-api/test-api.jar"]

While I solved the questioned problem, this script has a little problem: there will be 2 separate Gradle dependencies downloads(Test and Build). I couldn't manage to use Cloud SQL Proxy on gcr.io/cloud-builders/docker, so I workaround by using the Test step instead of the Build step. Maybe this can be solved using either docker run --network="host" or host.docker.internal, but I didn't try.

DB connection during integration tests works on local, but not working on Google Cloud Build

1 Answers1