3

We are using KubeDB in our cluster to manage our DB's.

So Redis is deployed via a KubeDB Redis object and KubeDB attaches a PVC to the Redis pod.

Unfortunately KubeDB doesn't support any restoring or backing up of Redis dumps (yet).

For the backup our solution is to have a CronJob running which copies the dump.rdb from the Redis pod into the job pod and then uploads it to S3.

For the restoring of the dump I wanted to do the same, just the other way around. Have a temporary pod which downloads the S3 backup and then copies it over to the Redis pod into the dump.rdb location.

The redis.conf looks like this:

....
# The filename where to dump the DB
dbfilename dump.rdb

# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /data
....

The copying works. The dump.rdb is in the correct location with the correct permissions. I verified this by starting a second redis-server in the Redis pod using the same redis.conf. The dump.rdb is being loaded into the server without a problem.

However, since I don't want to manually start a second redis-server, I restarted the Redis pod (by kubectl delete pods) for the pod to pickup the copied dump.rdb.

Everytime I delete the pod, the dump.rdb is deleted and a new dump.rdb is being created with a much smaller size (93 bytes).

I don't believe it is a PVC issue since I have created a few files to test whether they are deleted as well. They are not. Only the dump.rdb.

Why does this happen? I am expecting Redis to just restore the DB from the dump.rdb and not create a new one.

EDIT: Yeah, size of dump.rdb is around 47 GB. Redis version is 4.0.11.

Sajib Khan
  • 22,878
  • 9
  • 63
  • 73
Moritz Schmitz v. Hülst
  • 3,229
  • 4
  • 36
  • 63
  • Hi. KubeDB dev here. Can you try using `dir /data/` (note the trailing `/` ) in `redis.conf` file? Also, how did you provided `redis.conf` file to KubeDB? – Emruz Hossain Jun 28 '19 at 12:28
  • 1
    Hey man, thanks for helping. `redis.conf` is mounted as `configSource.configMap.name: redis-configmap`. But I think we discovered the problem already... Redis automatically dumps the current data on [shutdown](https://redis.io/commands/shutdown). So the fix is to restart the pod by `kubectl exec redis-pod redis-cli SHUTDOWN NOSAVE`. – Moritz Schmitz v. Hülst Jun 28 '19 at 14:08

2 Answers2

7

Sooo, a few hours later, my teammate remembered that Redis executes a save to dump on shutdown.

Instead of deleting the pod using kubectl delete pod I now changed the code to run a SHUTDOWN NOSAVE using the redis-cli.

kubectl exec <redis-pod> -- /bin/bash -c 'redis-cli -a $(cat /usr/local/etc/redis/redis.conf | grep "requirepass " | sed -e "s/requirepass //g") SHUTDOWN NOSAVE'
Moritz Schmitz v. Hülst
  • 3,229
  • 4
  • 36
  • 63
2

Restore Redis on Kubernetes AOF = yes:

The first thing to do is remove redis deployment from kubernetes server:

kubectl delete -f ./redis.yaml

Attach to the redis persistent storage (PVC) on mounted file system it can be GlusterFS - Volume, Azure Storage - File Share, min.io S3 bucket

Then, remove the current dumb.rdb file (if there is one) or rename it to dump.rdb.old:

Copy the good backup dump.rdb file in and correct its permission:

chown 999:999 dump.rdb
chmod 644 dump.rdb

Next the important part is to disable AOF by editing redis.yaml file, set appendonly as "no": Verify appendonlu is set to "no":

  containers:
    - name: redis
      image: redis:5.0.4
      imagePullPolicy: Always
      args: ["--requirepass", "$(redis_pass)", "--appendonly", "no", "--save", "900", "1", "--save", "30", "1"]

Next create the Redis deployment on kubernetes:

kubectl apply-f ./redis.yaml

Run the following command to create new appendonly.aof file

kubectl exec redis-0 -- redis-cli -a <redis-secret> bgrewriteaof

Check the progress (0 - done, 1 - not yet), and if exists new appendonly.aof file on the same size like dump.rdb

kubectl exec redis-0 -- redis-cli -a <redis-secret> info | grep aof_rewrite_in_progress

You should see a new appendonly.aof file. Next, recreate redis server: After it finished, enable AOF again by changing redis.yaml file to yes

  containers:
    - name: redis
      image: redis:5.0.4
      imagePullPolicy: Always
      args: ["--requirepass", "$(redis_pass)", "--appendonly", "yes", "--save", "900", "1", "--save", "30", "1"]

Then recreate the Redis server again:

kubectl delete-f ./redis.yaml
kubectl apply-f ./redis.yaml

The Restore is completed.

If you have linux with installed redis as service please use this instruction: https://community.pivotal.io/s/article/How-to-Backup-and-Restore-Open-Source-Redis?language=en_US

rafalkasa
  • 1,743
  • 20
  • 20