0

I have seen a lot of questions and topics but they are all related to docker-compose and creation of kafka container. But I have a 1 namenode and 3 datanode container system. I have two ports related to docker container, one is 8088 and the other is 50070. I want to send data using kafka from my local to the docker. But for me I need to send the data using 8088 or 50070. But I can't figure out how to do this. I edited

listeners=SASL_PLAINTEXT://0.0.0.0:8088, PLAINTEXT://0.0.0.0:9092 advertised.listeners=SASL_PLAINTEXT://localhost:8088, PLAINTEXT://localhost:9092 security.inter.broker.protocol=SASL_PLAINTEXT

in server.properties but it didn't work. I am trying like this: I created a topic named test in Docker. I sent some data to it on Docker terminal. And in local I try to consume the data from topic with --bootstrap-server localhost:8088 but it gives error. Thank you.

Yunus Emrah Uluçay
  • 129
  • 9

1 Answers1

0

have a 1 namenode and 3 datanode container system

These are not Kafka services, and Docker is irrelevant to the question.

If you want to send data to Hadoop datanodes on these ports, you need to use Kafka Connect, or from Python you can use Spark, Flink, etc.

If all services are running in containers, then Compose or your host isn't completely relevant, either.

You'd use a bridge network - https://docs.docker.com/network/bridge/

I need to send the data using 8088 or 50070.

Ports don't matter, but you really should consider leaving Kafka ports as the defaults.

But you must forward the listener port from the host, properly -p 8088:8088, and you must set the necessary SASL properties in whatever command you're using with --bootstrap-server localhost:8088, which should work from both the host and inside the container (which is where you should first debug the problem)

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Yes you are right the system of my docker is irrelevant for this question. From your answer I understand I need to create a Docker compose. "I need to send the data using 8088 or 50070." here I mean forward the Kafka's port to these ports to communicate with my local but I couldn't make it. For example I create a topic in container and sent some data to it. After that when I try to get data using consumer from outside the docker I can not connect the topic. – Yunus Emrah Uluçay Jun 16 '22 at 10:01
  • 1
    Compose is not a requirement, it just makes it easier to deploy multiple linked containers. Again, ports shouldn't need to be changed or remapped. Please read this to solve your problem https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker#51634499 However, this still doesn't address why you're trying to use **Hadoop port values** for Kafka communication – OneCricketeer Jun 16 '22 at 12:42