0

My task is to run Kafka connectors in docker containers, when I tried run connectors on my host all worked fine, but in container it fails. I know that there are a lot similar questions( I am new in Kafka and Redis and really can not find solution). I tried a lot solutuons, the algorithm from this answer Connecting a Redis container with another container (Docker) seems really amazing, but in doesn't work for me and I can't understand why.

In Docker app it gives me stack trace as:

ERROR WorkerSinkTask{id=kafka-connect-redis-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:186) io.lettuce.core.RedisConnectionException: Unable to connect to redis-master:6379 at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: redis-master/172.22.0.3:6379 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

My Redis config:

replicaof redis-master 6379
# bind 0.0.0.0
protected-mode no

My connector properties file

name=kafka-connect-redis
topics=bots
tasks.max=1
connector.class=com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector
key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.storage.StringConverter
#offset.storage.file.filename=/tmp/connection/redis/connect.offsets
redis.hosts=redis-master,redis-slave-1,redis-slave-2
use.record.key=true

My docker compose:

   redis-master:
     image: redis
     container_name: redis-m
#    ports:
#      - 6379:6379 
     volumes:
       - ./clear-redis.sh:/bin/clear-redis.sh
     command: bash -c "chmod +x /bin/clear-redis.sh && bash /bin/clear-redis.sh"

  redis-slave-1:
    image: redis
    container_name: redis-s-1
#    ports:
#      - 7000:6379
    volumes:
      - ./config/redis:/usr/local/etc/redis/
    command: redis-server /usr/local/etc/redis/redis.conf

  redis-slave-2:
    image: redis
    container_name: redis-s-2
#    ports:
#      - 7001:6379
    volumes:
      - ./config/redis:/usr/local/etc/redis/
    command: redis-server /usr/local/etc/redis/redis.conf

   
  kafka-connect-redis:
    image: confluentinc/cp-kafka-connect:5.5.1
    container_name: connect-redis
    hostname: kafka-connect-redis
    ports:
      - 8086:8086
    volumes:
      - ./connection/connectors/kafka-connect-redis/lib/:/etc/kafka-connect/jars/
      - ./connection/connectors/kafka-connect-redis/redis-sink.properties:/usr/share/redis-sink.properties
      - ./connection/connect-standalone-2.properties:/etc/connect-standalone-2.properties
      - ./connection/run-redis.sh:/bin/run-redis.sh
    depends_on:
      - zoo
      - kafka1
      - kafka2
      - kafka3
      - redis-master
      - redis-slave-1
      - redis-slave-2
    command: bash -c "chmod +x /bin/run-redis.sh && ./bin/run-redis.sh"
kalif
  • 11
  • 2
  • Did you try to uncomment the bind line in the Redis properties? Have you tried to ping the Redis containers from within the Connect container? Also, start simpler - use only one Redis instance – OneCricketeer May 11 '21 at 22:49
  • 1) Yes, I tried, no special, all the same. 2) Yes, `ping redis-master` inside kafka-connect-redis container works 3) I tried use only master node, and commented all lines that tied to redis master. I also have another connector working with files and it works fine in docker container. So, I think the problem is in my config files, but where is, I can't understand. – kalif May 12 '21 at 06:38
  • I don't think I've ever used the Redis containers, but "Connection refused" has clear symptoms. Since the DNS between containers works, that means that the server either hasn't opened the port you've told it to (you can check that with netcat from another container) or the bind address didn't accept a remote connection (which it might not if the bind address isn't set. For example, you're not mounting any config file to the master – OneCricketeer May 12 '21 at 12:31

0 Answers0