How to configure redis on GCP using cloudbuild.yaml?

Question

I'm trying to setup a CI/CD pipeline on GCP. I have a nodejs application which uses redis as a database. I am trying to configure redis on GCP.

I have tried below configuration but once the redis step is executed, it keeps waiting for the redis connection and doesn't go the next step and after some time, build timeout's.

Cloudbuild.yaml file:

steps: - name: 'gcr.io/cloud-builders/npm' args: ['install'] - name: 'redis' env: ['REDISHOST=127.0.0.1', 'REDISPORT=6379'] - name: 'gcr.io/cloud-builders/npm' args: ['test'] - name: "gcr.io/cloud-builders/gcloud" args: ["app deploy"]

Error log:

ERROR: context deadline exceeded TIMEOUT Finished Step #1 Step #1: 1:M 06 Sep 2019 18:43:09.317 # Redis is now ready to exit, bye bye... Step #1: 1:M 06 Sep 2019 18:43:09.317 * DB saved on disk Step #1: 1:M 06 Sep 2019 18:43:09.312 * Saving the final RDB snapshot before exiting. Step #1: 1:M 06 Sep 2019 18:43:09.312 # User requested shutdown... Step #1: 1:signal-handler (1567795389) Received SIGTERM scheduling shutdown... Step #1: 1:M 06 Sep 2019 18:33:38.595 * Ready to accept connections Step #1: 1:M 06 Sep 2019 18:33:38.595 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. Step #1: 1:M 06 Sep 2019 18:33:38.595 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. Step #1: 1:M 06 Sep 2019 18:33:38.595 # Server initialized Step #1: 1:M 06 Sep 2019 18:33:38.595 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. Step #1: 1:M 06 Sep 2019 18:33:38.594 * Running mode=standalone, port=6379. Step #1: 1:C 06 Sep 2019 18:33:38.592 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf Step #1: 1:C 06 Sep 2019 18:33:38.592 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=1, just started Step #1: 1:C 06 Sep 2019 18:33:38.592 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo Step #1: docker.io/library/redis:latest Step #1: Status: Downloaded newer image for redis:latest Step #1: Digest: sha256:0e67625224c1da47cb3270e7a861a83e332f708d3d89dde0cbed432c94824d9a Step #1: 93e8c2071125: Pull complete Step #1: 8e4f9890211f: Pull complete Step #1: 8a9a85c968a2: Pull complete Step #1: c1b01f4f76d9: Pull complete

Answer 1

In Google Cloud Build, each step is executed in a separate Docker container, one after the other. In your case, the second step starts the redis container which waits for connections. The build is "stuck" and times out.

In this situation, you may run the Redis container in the background using Docker compose. Your cloudbuild.yaml file may look something like this:

steps:
- name: 'gcr.io/cloud-builders/npm'
  args: ['install']
- name: 'docker/compose:1.24.1' # you can use the version of your choice
  args: ['up', '-d']
- name: 'gcr.io/cloud-builders/npm'
  args: ['test']
  env:
  - 'HOST=redis' # name of the running container
  - 'PORT=6379'
- name: "gcr.io/cloud-builders/gcloud"
  args: ['app', 'deploy'] # NOTE THAT YOU HAVE TO PROVIDE THIS AS 2 PARAMETERS

Your docker-compose.yml file may look something like this:

version: '3'
services:
  redis:
    image: redis
    network_mode: cloudbuild
    container_name: redis
    expose:
      - 6379

Note the network_mode: cloudbuild configuration. As documented here: "Each build step is run with its container attached to a local Docker network named cloudbuild". We instruct Docker Compose to run the redis container in this network so that they can communicate.

Answer 2

Did some googling and I came up with a few other stack overflow cases that all deal with the warning messages in your error. Specifically, these 3 messages:

Step #1: 1:M 06 Sep 2019 18:33:38.595 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

Step #1: 1:M 06 Sep 2019 18:33:38.595 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

Step #1: 1:M 06 Sep 2019 18:33:38.595 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

Here are the cases I found:

When to turn off Transparent Huge Pages for redis

Solving Redis warnings on overcommit_memory and Transparent Huge Pages for Ubuntu 16.04 on EC2

What's "tcp-backlog" in redis.conf

your issue is probably related to having Transparent Huge Pages (THP) turned on.

Follow the answer from the 'When to turn off Transparent Huge Pages for redis' article and you should be good to go. Also, here is the latency troubleshooting guide from redis.

https://redis.io/topics/latency