39

Ive got a local kafka running using the following docker-compose.yml

version: '2'
services:
  zookeeper:
    image: "confluentinc/cp-zookeeper:5.0.1"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: "confluentinc/cp-enterprise-kafka:5.0.1"
    ports:
      - '9092:9092'
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100

Trying to run a basic create topic using kafka-client 2.1.0 in Scala:

val props = new Properties()
props.setProperty(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")

val adminClient: AdminClient = AdminClient.create(props)
val newTopic = new NewTopic("test", 1, 1.toShort)
val topicsF = adminClient.createTopics(List(newTopic).asJavaCollection)
val result = topicsF.all().get()

but after some time I get: 

org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

I can create a topic using the command line:

kafka-topics --create \
    --zookeeper localhost:2181 \
    --replication-factor 1 \
    --partitions 1 \
    --topic test
Created topic "test".

kafka AdminClient API Timed out waiting for node assignment describes a similar problem using Java but the comment suggests that a system restart fixed the issue which is not the case on my side.

Milan
  • 929
  • 2
  • 13
  • 25
  • for future reference: for those looking at this question and wondering why the environment looks the same in the question and the answer - @Milan edited the question after the answer, see the question history – dyang May 19 '20 at 21:58

6 Answers6

42

If you're running Kafka in Docker (or similar) you need to configure the listeners correctly. This article describes it in detail.

Here's an example of a Docker Compose that you can use to access Kafka from your host machine.

Disclaimer: I wrote the article :)

lloiacono
  • 4,714
  • 2
  • 30
  • 46
Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • I have updated my question with a docker-compose.yml file. With this setup, it still does not work. – Milan Dec 03 '18 at 11:20
  • where are you running your Scala code? on the Docker host or on another container? – Robin Moffatt Dec 03 '18 at 13:59
  • I start the described docker-compose.yml and run the code locally from my MacOS as I just want to quickly develop without the need to build an sbt container and deploy it. – Milan Dec 03 '18 at 14:02
  • I'm not familiar with the Docker image you're using. I've updated my answer with some additional material – Robin Moffatt Dec 03 '18 at 14:05
  • I have updated my question with the proposed docker-compose.yml. But when I run the simple code, it still fails with the same error. – Milan Dec 03 '18 at 14:15
  • Does it work if you use `kafkacat` as described in the article I link to? – Robin Moffatt Dec 03 '18 at 15:46
  • This link is not working: https://github.com/confluentinc/demo-scene/blob/master/cos/docker-compose.yml#L5-L28 – lloiacono Feb 11 '19 at 10:46
  • 1
    @lloiacono fixed: https://github.com/confluentinc/demo-scene/blob/master/community-components-only/docker-compose.yml#L22-L51 – Robin Moffatt Feb 11 '19 at 11:05
12

As @suh pointed out.

An easier approach is uncomment kafka server.properties at line :listeners=PLAINTEXT://localhost:9092 and it should work.

Nakamoto
  • 1,293
  • 14
  • 17
5

the main thing you have to do is

  • first open your kafka server.properties from this path

    nano kafka-directory/config/server.properties

  • now go to change this line. listeners=PLAINTEXT://:9092 with this

    listeners=PLAINTEXT://your ip address:9092

your ip address must be an IP of the same network with the machine where you attempt to connect to kafka i mean if the ip of the client machine is late say 192.168.10.1 your listeners ip configuration must be as:

listeners=PLAINTEXT://192.168.10.*:9092

  • next modifie this configuration advertised.listeners= to this

    advertised.listeners=PLAINTEXT://your ip address:9092

thes two configurations must have the same IP and port at the end you got some thing like.

listeners=PLAINTEXT://192.168.10.10:9092
advertised.listeners=PLAINTEXT://192.168.10.10:9092
Mukendi Emmanuel
  • 121
  • 1
  • 7
2

I think the localhost is the problem. In your bootstrap-servers properties use the advertised host (192.168.99.100) that you've defined in your compose file, instead of localhost, that should work.

Bitswazsky
  • 4,242
  • 3
  • 29
  • 58
  • 1
    Can you look inside the container, what's the advertised host defined in the server.properties? Then you can try doing a telnet to 9092 from outside, and see if it's able to connect. – Bitswazsky Dec 03 '18 at 04:39
  • Inside of the container in server.properties: advertised.listeners=PLAINTEXT://192.168.99.100:9092. > telnet 192.168.99.100:9092 192.168.99.100:9092: nodename nor servname provided, or not known – Milan Dec 03 '18 at 05:32
  • I have updated my question with a docker-compose.yml file. With this setup, it still does not work. – Milan Dec 03 '18 at 11:20
  • 3
    For me changing "listeners=PLAINTEXT://:9092" to "listeners=PLAINTEXT://localhost:9092" in the kafka server.properties worked. Thanks! – suh May 22 '20 at 15:23
0

My advertised listeners were wrong and i did not have KAFKA_LISTENER_SECURITY_PROTOCOL_MAP configured.

A good example i found here:

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,SSL:SSL,SSL_HOST:SSL,SASL_SSL:SASL_SSL,SASL_SSL_HOST:SASL_SSL
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092,SSL://kafka1:29093,SSL_HOST://localhost:9093,SASL_SSL://kafka1:29094,SASL_SSL_HOST://localhost:9094

P.S: Use only what you need, for me it was SASL_SSL & SASL_SSL_HOST.

jumping_monkey
  • 5,941
  • 2
  • 43
  • 58
0

Configuring using local Kafka. un comment change and the listeners PLAINTEXT to

listeners=PLAINTEXT://127.0.0.1:9092

also, change the create topic command bootsrap-server to point to the plain text server address.

kafka-topics.bat --create  --bootstrap-server 127.0.0.1:9092 --replication-factor 1  --partitions 1 --topic TestTopic

Created topic TestTopic.

Muyinda Rogers
  • 147
  • 3
  • 13