Redis Sentinel Docker Containers Failing Immediately

Question

I am trying to set up a high availability Redis clusetr using Sentiner but I am having some issues with Sentinel and Docker.

I am using the below command to set up 3 clusters. The issue is that they are created and immediately fail. I see the hash from the creation, then they are gone.

Any idea what is causing them to fail? The matching Redis server instances stand up just fine.

Sentinel

#!/bin/bash

REDIS_IMAGE="redis/redis-stack:7.0.2-RC3"
REDIS_NET="redis"
NETWORKS=("$REDIS_NET" "api" "postgres")

function check_success {
    if [ $? -ne 0 ]
    then
        echo "[!] Command failed. Exiting"
        exit 1
    fi
}

if [ -z "$1" ]
then
  echo "Supply an argument: 'dev', 'prod'. All lowercase."
  exit 1
else
  ENV=$1
fi

if [ "$2" == "restart" ]
then
  echo "[-] Restart /usr/bin/docker.socket"
  sudo systemctl restart docker.socket
  check_success

  echo "[-] restart /usr/bin/docker.service"
  sudo systemctl restart docker.service
  check_success
fi



echo "[-] Stopping and removing existing containers and networks"
sudo /usr/bin/docker stop $(sudo /usr/bin/docker ps -a -q)
sudo /usr/bin/docker rm $(sudo /usr/bin/docker ps -a -q)
sudo /usr/bin/docker network prune
check_success

for net in "${NETWORKS[@]}"
do
  sudo docker network ls | grep "$net"
  # If no network called n create it
  if [ $? -ne 0 ]
  then
    echo "Creating network: $net"
    sudo /usr/bin/docker network create "$net"
    check_success
  fi
done


# Create redis instances
for i in $(seq 0 2)
do
  echo "[-] Create redis-$i"
  sudo /usr/bin/docker run -d --rm --name "redis-$i" \
      --net $REDIS_NET \
      -v "$(pwd)/redis/clustering/redis-$i":/etc/redis/ \
      $REDIS_IMAGE redis-server /etc/redis/redis.conf
  check_success
done

# Create sentinel instances
for i in $(seq 0 2)
do
  echo "[-] Create sentinel-$i"
  sudo /usr/bin/docker run -d --rm --name "sentinel-$i" \
      --net $REDIS_NET \
      -v "$(pwd)/redis/clustering/sentinel-$i":/etc/redis/ \
      $REDIS_IMAGE redis-sentinel /etc/redis/sentinel.conf
  check_success
done

# List instances
echo "[-] Docker networks"
sudo /usr/bin/docker network ls
check_success

sudo docker-compose -f docker-compose-$ENV.yml up --build

redis.conf for redis-0

protected-mode no
port 6379

#authentication
requirepass "some_secret"
masterauth super-very-secret-password-goes-here
tls-protocols "TLSv1.2 TLSv1.3"

# RDB
dbfilename dump.rdb
save 3600 1 300 100 60 10000

# AOF
appendonly yes
appendfilename "appendonly.aof"

This is my sentinel-0.conf

port 5000
sentinel monitor mymaster redis-0 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster super-very-secret-password-goes-here

EDIT

After removing the --rm option the following shows in the logs for sentinal-0

1:X 12 Nov 2022 22:49:14.563 # Failed to resolve hostname 'redis-0'

*** FATAL CONFIG FILE ERROR (Redis 7.0.4) ***
Reading the configuration file, at line 2
>>> 'sentinel monitor mymaster redis-0 6379 2'
Can't resolve instance hostname.

This is strange as all the sentinels and servers are on the same 'redis' network

The only other place the redis network is configured is in my compose file, but the containers fail before that runs.

networks:
  redis:
    driver: bridge
  api:
    driver: bridge
  postgres:
    driver: bridge

What do you see in the container logs? Drop the `--rm` from the command line so you can inspect them after they fail. — larsks, Nov 12 '22 at 22:48
Looks like it can't find the redis-n instances, even though it's on the same network and they are up — 3therk1ll, Nov 12 '22 at 22:52
If you could post the *complete script*, rather than excerpts, that would be helpful. — larsks, Nov 12 '22 at 23:00

score 1 · Answer 1 · answered Nov 12 '22 at 23:15

1

I figured it out from this issue here.

redis-sentinel throws error: " Can't resolve master instance hostname."

I just needed to add the following to allow Sentinel to resolve hostnames.

sentinel resolve-hostnames yes

answered Nov 12 '22 at 23:15

3therk1ll

2,056
4
35
64

I'm glad you figured it out! To forestall another question, note that your network `redis` in your `docker-compose.yaml` is not the same `redis` network you've used for your redis and sentinel containers -- you need to mark it as an external network by setting `external: true` (see https://docs.docker.com/compose/compose-file/compose-file-v3/#external-1) – larsks Nov 12 '22 at 23:53
Oh nice one, cheers buddy – 3therk1ll Nov 13 '22 at 00:00

Redis Sentinel Docker Containers Failing Immediately

1 Answers1