why kafka producer is showing me error kafka.conn:DNS lookup failed for :9092?

Question

I expose the port 9092 then i run the kafka broker inside docker. But When I run the python script i get the errors

ERROR:kafka.conn:DNS lookup failed for b5c5b06f6761:9092 (AddressFamily.AF_UNSPEC)

I tried docker ip and machine ip instead of localhost but gives same error.

Here is my code.

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],
                         value_serializer=lambda x:
                         dumps(x).encode('utf-8'))

producer.send('vtintel', value={'id':123})

i suspect you need to add something in your hosts file (if you use windows) — Gremi64, May 05 '19 at 15:40
docker run -p 8081-8110:8081-8110 -p 8200:8200 -p 9095:9095 -p 9097:9097 -p 9999:9999 -p 9092:9092 -d --name imply imply/imply — Hiba Rehman, May 05 '19 at 15:41
@Gremi64 That's a hack, please never do that and just fix the actual problem... https://rmoff.net/2018/08/02/kafka-listeners-explained/ — OneCricketeer, May 05 '19 at 17:05
@cricket_007 i know, and you are right to point it out. I thought (don't know why) that he was doing this as a "local test" — Gremi64, May 05 '19 at 17:33
you'll find your answer in @cricket_007 link, or in this thread https://stackoverflow.com/questions/52438822/docker-kafka-w-python-consumer/52440056#52440056 or this one https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker-from-local-machine — Gremi64, May 05 '19 at 18:39

score 7 · Answer 1 · edited Oct 13 '20 at 20:11

Same issue with bitnami/kafka. But then I realized that I need to enable accessing Kafka with external clients in docker-commpose.yml. For more info see https://hub.docker.com/r/bitnami/kafka/ 'Accessing Kafka with internal and external clients' part.

To do so, add the following environment variables to your docker-compose:

    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
+     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+     - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
+     - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

And expose the extra port:

    ports:
      - '9092:9092'
+     - '29092:29092'


and access it by 'localhost:29092' not 'localhost:9092' in your python-kafka code.

Incase it's not obvious, that should be added to the kafka container (not the zookeeper container)

score 2 · Answer 2 · answered May 05 '19 at 17:12

Docker only handles DNS within its own network, not from your host

You need to Kafka to advertise itself externally (on localhost), which is different than just a port forward

And as far as I can tell -p 9092:9092 is not a port even exposed by the container image you're using

score 2 · Answer 3 · answered Oct 24 '19 at 13:31

Though it's a late reply it might help anyone later. I have the same problem as you were facing I was using dockerized implementation of edX. So to fix that issue just add the following lines in
/etc/hosts
file of your docker container. first your IP Address then the thing for which the lookup is being failed. for example in your case lookup is being failed for b5c5b06f6761 so:

173.16.18.22     b5c5b06f6761

Note: Here I am using dummy IP address.

score -1 · Answer 4 · answered May 05 '19 at 17:02

-1

I had a similar problem earlier with latest kafka versions. Try mentioning the local address as '127.0.0.1' instead of 'localhost'. this might help.

answered May 05 '19 at 17:02

Sameer Killamsetty

47
5

why kafka producer is showing me error kafka.conn:DNS lookup failed for :9092?

4 Answers4