Kafka - Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092

Question

I'm trying to get one of my services up and running locally within the docker network, but keep getting "Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092" exception.

zookeeper:
  image: wurstmeister/zookeeper:3.4.6
  restart: always
  expose:
      - "2181"



kafka:
      depends_on:
          - zookeeper
      environment:
          KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
          KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      hostname: kafka
      image: wurstmeister/kafka:2.11-2.0.0
      restart: always
      ports:
          - "9092:9092"
      expose:
          - "9093"
      links:
          - zookeeper:zookeeper

That's what I've set in my service's docker-compose for the consumers:

  consumers:
    environment:
      - BROKER_URL=host.docker.internal:9092

That's the error message:

..`block in execute'","/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'"],
"message":"Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092"},
"message":"Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092",
"@timestamp":"2020-05-24T18:08:09.434+00:00","@version":"1",
"severity":"ERROR","host":"87c5265c6a39"}

UPDATE: When checking Kafkacat:

kafkacat -L -b 127.0.0.1:9092
Metadata for all topics (from broker 0: 127.0.0.1:9092/0):
 1 brokers:
  broker 0 at 127.0.0.1:9092 (controller)
 7 topics: ...

When checking Zookeeper CLI (within Docker):

[zk: localhost:2181(CONNECTED) 2] ls /brokers/ids
[1001]

I've tried already pretty much everything I could find, but still not able to understand what am I missing.

Any help would be appreciated, thanks!

Where's your consumer running? Is it on the Docker network? or somewhere else? — Robin Moffatt, May 24 '20 at 20:40
@RobinMoffatt Forgot to mention, it's within the docker network. — Lior G., May 24 '20 at 20:54
Your consumer should connect to `kafka:9093` in the same docker network. Also `links` are a deprecated feature — OneCricketeer, May 26 '20 at 15:57

score 1 · Answer 1 · answered May 25 '20 at 06:33

1

You’re connecting to the wrong listener. When you connect to 9092 the broker returns localhost for your client to subsequently connect to. Connect to 9093 and it will use kafka which your consumer in the Docker network should be able to resolve.

Ref: https://rmoff.net/2018/08/02/kafka-listeners-explained/

answered May 25 '20 at 06:33

Robin Moffatt

30,382
3
65
92

Yes, This! I had similar problem and was able to use this hint:" INSIDE://kafka:9093. ". In my case is "KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092". Changed the producer code to connect to "broker:29092" – Igor Ostaptchenko Feb 07 '23 at 15:19

Kafka - Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092

1 Answers1