Unable to connect to Kafka run in container from Spring Boot app run outside container

Question

I'm running kafka locally via:

docker-compose.yml

  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - 2181:2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - 9092:9092
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092

My Spring Boot application is run with:

application.yml:

spring:
  application:
    name: testkafka
  kafka:
    bootstrap-servers: localhost:9092

server:
  port: 8080

When I run it and it tries to send to a topic on kafka, I get:

org.springframework.kafka.KafkaException: Reply timed out
    at org.springframework.kafka.requestreply.ReplyingKafkaTemplate.lambda$sendAndReceive$0(ReplyingKafkaTemplate.java:196) ~[spring-kafka-2.1.10.RELEASE.jar:2.1.10.RELEASE]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.10.RELEASE.jar:5.0.10.RELEASE]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

If I run the spring boot from inside a docker container (using one docker compose file), then it does work:

Running Both in the Same Compose:

version: "3.0"
services:
  service1:
    build: ./Service
    ports:
      - 8080:8080
      - 5005:5005
    links:
      - zookeeper
      - kafka
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - 2181:2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - 9092:9092
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092

How can I get the kafka container to allow connections from outside itself/docker?

EDIT: Tried these changes:

kafka:
  image: 'bitnami/kafka:latest'
  ports:
    - 9092:9092
  environment:
    - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
    - ALLOW_PLAINTEXT_LISTENER=yes
    - KAFKA_ADVERTISED_PORT=9092
    - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092

and:

spring:
  application:
    name: testkafka
  kafka:
    bootstrap-servers: kafka:9092

server:
  port: 8080

This still times out

This question is asked rather repeatedly. Please read https://rmoff.net/2018/08/02/kafka-listeners-explained/ — OneCricketeer, Nov 01 '18 at 00:21

OneCricketeer · Accepted Answer · 2021-02-11T18:00:21.373

10

If I run the spring boot from inside a docker container (using one docker compose file), then it does work

It shouldn't work, actually. Kafka isn't running as part of the application, so this section is not pointing at the Kafka container.

kafka:
    bootstrap-servers: localhost:9092

It would need to be kafka:9092 within the Docker network.

and ... bootstrap-servers: kafka:9092 ... still times out

That's because you still have KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092

Both inside and outside of Docker network, by specifying KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092, you're saying that your clients receive the Kafka bootstrap connection as localhost:9092, which would work outside of the Docker network only because you have made both a port forwarding and your container is running on localhost, however, inside the Docker network, as mentioned, localhost would mean that application container, not the broker.

The solution would be to create two port mappings via that property, as discussed in length by this blog post

Plus, Confluent provides a fully configured Docker Compose with appropriate mappings that'll work inside and out of Docker

    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://0.0.0.0:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

For applications within Docker network, use kafka:9092, for applications outside, use localhost:29092

EDIT The bitnami config variables have changed. Refer this section of the README

https://github.com/bitnami/bitnami-docker-kafka/blob/master/README.md#accessing-kafka-with-internal-and-external-clients

edited Feb 11 '21 at 18:00

answered Nov 01 '18 at 00:28

OneCricketeer

179,855
19
132
245

in addition to Cricket answer you can then add `kafka` in your host file and connect to it from out side the docker-compose, given `KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092`, `bootstrap-servers: kafka:9092` and `ports - 9092:9092` – Paizo Nov 01 '18 at 14:16
@Paizo Editing the hosts file is a hack, and should be avoided. Correcting the listeners is the correct thing. – OneCricketeer Nov 01 '18 at 16:42
@DonRhummy See updated answer. The gist here is that you can't have the same spring config working inside and outside of Docker – OneCricketeer Nov 01 '18 at 16:45
@cricket_007 I get the error `java.lang.IllegalArgumentException: requirement failed: advertised.listeners listener names must be equal to or a subset of the ones defined in listeners. Found PLAINTEXT,PLAINTEXT_HOST. The valid options based on the current configuration are PLAINTEXT` – Don Rhummy Nov 01 '18 at 19:34
@DonRhummy Have you defined `KAFKA_LISTENER_SECURITY_PROTOCOL_MAP`, as mentioned in my answer? – OneCricketeer Nov 01 '18 at 20:08
@cricket_007 yes. I believe the issue is with the bitnami image. Can you try it yourself? – Don Rhummy Nov 01 '18 at 21:22
@DonRhummy Fixed. See updated answer. The consumer seemed to kill the container on the first request, but restarting the container, I saw the messages – OneCricketeer Nov 01 '18 at 23:02
@cricket_007 If you can add `ALLOW_PLAINTEXT_LISTENER: "yes"` I'll mark this as the answer. Without that it fails. – Don Rhummy Nov 02 '18 at 00:02

Unable to connect to Kafka run in container from Spring Boot app run outside container

1 Answers1

Linked