0

I just wanted to implement Spring boot and Apache Kafka via docker-compose. Firstly I installed the Apache Kafka Docker image via this link: https://hub.docker.com/r/ches/kafka/ and then install jplock/zookeeper via this command shown below.

docker run -d — name zookeeper — publish 2181:2181 jplock/zookeeper:latest

Then I run this command (docker ps) to determine if all images work. their status of all these images is up. That's why they work flawlessly.

Images

Then I run the app and sent a message via Postman. The post url is localhost:8080/kafkamessage.

JSON object is here

{
    "message" : "Hello World"
}

When I sent a request, it throws an error shown below.

"message": "Send failed; nested exception is org.apache.kafka.common.errors.TimeoutException: Topic k-topic not present in metadata after 60000 ms."

Expect for this, I defined an auto-created topic but the consumer and producer couldn't produce the topic in the console

[Consumer clientId=consumer-k-group-1, groupId=k-group] Error while fetching metadata with correlation id 728 : {k-topic=LEADER_NOT_AVAILABLE}
[Producer clientId=producer-1] Error while fetching metadata with correlation id 681 : {k-topic=LEADER_NOT_AVAILABLE}

Here is my error

enter image description here

Here is the link: https://github.com/Rapter1990/springbootkafka

How can I fix the issue?

enter image description here

S.N
  • 2,157
  • 3
  • 29
  • 78
  • Please don't use Spotify image. It's not maintained and uses very old version of Kafka – OneCricketeer Feb 11 '21 at 15:04
  • @OneCricketeer Did the problem trigger to throw an error because of the usage of Spotify. I don't think so. – S.N Feb 11 '21 at 17:05
  • It might have if `ADVERTISED_HOST` and `ADVERTISED_PORT` are not the correct variables to configure it with (and both of those are deprecated Kafka configurations anyway). Besides, that, you've overrwritten the `command` of the container, and so its not clear if the Kafka (or Zookeeper) processes actually started... If you need to create a topic, then do that in Spring with a `@Bean NewTopic` – OneCricketeer Feb 11 '21 at 17:30
  • @OneCricketeer what variables should I use instead of ADVERTISED_HOST and ADVERTISED_PORT in docker-compose.yml? I think the topic generates automatically because of AUTO_CREATE_TOPICS. Can you look through my screenshots if you don''t mind? I want to know where the issue is. – S.N Feb 11 '21 at 17:39
  • Only kafka image that can create topics with an env-var is the wurstmeister image. Auto topic creation is generally disabled in most Kafka environments. You should refer to the linked duplicate question to address your issue or read https://www.confluent.io/blog/kafka-listeners-explained/ – OneCricketeer Feb 11 '21 at 17:47
  • Not clear how you're running the Spring app, either, but ideally, you'd also put it in the Compose file as another service. In which case, your post is similar to https://stackoverflow.com/questions/53089486/unable-to-connect-to-kafka-run-in-container-from-spring-boot-app-run-outside-con – OneCricketeer Feb 11 '21 at 17:56
  • @OneCricketeer All my screenshots are right, aren't they? Can you check my project? I already mentioned hot to run my Spring app. Can you help me how to rewrite docker-compose.yml? – S.N Feb 11 '21 at 17:58
  • I see no screenshots. You mean code? And you would add `servives: app: image: build: ./path/to/Dockerfile`. Other than that, I suggest you use one of the Kafka and Zookeeper Docker images from the questions I linked to (which also address the error you are getting) – OneCricketeer Feb 11 '21 at 18:02
  • @OneCricketeer All these are located under the docker_images folder. – S.N Feb 11 '21 at 18:05
  • I looked, but please [edit] your question to include screenshots rather than external links to a Github repo... In any case, why are you are using `docker run` commands? Do you not understand what `docker-compose` does, or why do you have that file if youre not using it? – OneCricketeer Feb 11 '21 at 18:07
  • @OneCricketeer I found this way to implement it. I added a link showing all containers in docker. As I show the issues, how can I fix the issue? Can you add a post showing the solution? – S.N Feb 11 '21 at 18:22
  • Can you explain what is unclear from the other two posts I linked to and the Confluent blog post? Do I need to provide more links? If I answer here, it just repeats all that information. – OneCricketeer Feb 11 '21 at 18:41
  • @OneCricketeer. I want to show all running containers in docker via this command (docker ps). After that, I run the Spring boot app, then it shows Consumer and Producer logs which contain LEADER_NOT_AVAILABLE. Lastly, I send a message from Postman and I got an error defined in my post. How can I fix it? – S.N Feb 11 '21 at 18:44
  • Once again. Read the listed posts... The answer you want is that your `ADVERTISED` variable given to the Kafka image is not using the same value as `k.kafka.address` in your Spring config. Make those have the same IP, and your issue should go away as long as you are not running your Spring code in a container – OneCricketeer Feb 11 '21 at 18:48
  • @OneCricketeer I changed it as "ADVERTISED_HOST=127.0.0.1" and I tested it but it didn't work. – S.N Feb 11 '21 at 18:54
  • The variable is `KAFKA_ADVERTISED_HOST_NAME`, as mentioned in the [Docker hub page](https://hub.docker.com/r/ches/kafka/) – OneCricketeer Feb 11 '21 at 18:55
  • @OneCricketeer I changed all enviromental variables but it still didn't work. – S.N Feb 11 '21 at 19:03
  • @OneCricketeer I edited my post. Firstly I stop this container and write the command but Term1 not work. – S.N Feb 11 '21 at 19:26

1 Answers1

1

Term1

docker run --rm --name zookeeper -p 2181:2181 jplock/zookeeper

Term2

(Where your error is)

docker run --rm -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=127.0.0.1 --link zookeeper:zookeeper ches/kafka

Note: ches/kafka has had no modifications since 2017 and is not maintained with latest improvments in Kafka

Term3

mvn clean package
java -jar ./target/springbootkafka-0.0.1-SNAPSHOT.jar

Term4

curl -X POST -H 'Content-Type: application/json' --data '{"message" : "Hello World"}' localhost:8080/kafkamessage

The producer worked, and so does a consumer

$ kafkacat -C -t k-topic -b localhost:9092 -o beginning
{"message":"Hello World","id":"ef0e7883-640c-473b-9b3d-7919d59594a1","messageDate":[2021,2,11,13,10,30,11737000]}
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I want to use Postman to send a request so how can I changed the term4 part? – S.N Feb 11 '21 at 19:18
  • 1
    Well it's a `POST` request with request body of type `application/json` containing the request body `{"message" : "Hello World"}`. The url is `localhost:8080/kafkamessage` this should suffice to transfer your request to Postman. – jAC Feb 11 '21 at 19:21
  • I cannot implement term1. The error is that docker: Error response from daemon: Conflict. The container name "/zookeeper" is already in use by container "b70ebdb369dd98987f95c3bf4897a5d1afaa653ccd7ae49a80f6566b5bc5ea0f". You hav e to remove (or rename) that container to be able to reuse that name. – S.N Feb 11 '21 at 19:25
  • @Michael You already have the container of the same name running. If you want to start fresh, then stop all other running containers.. Otherwise, skip it – OneCricketeer Feb 11 '21 at 19:26
  • @OneCricketeer I removed all of them and now try to implement your command in your way. – S.N Feb 11 '21 at 19:31
  • @OneCricketeer It works. Thank you. What if I could use local machine docker ip address instead of 127.0.0.1. How can I learn its ip address? – S.N Feb 11 '21 at 19:45
  • You shouldn't need that since youre using Docker Desktop – OneCricketeer Feb 11 '21 at 19:52
  • @OneCricketeer I don't use it as I use Docker Desktop. If not, I should learn ip address of zookeeper then write it KAFKA_ADVERTISED_HOST_NAME= ip address. Is it right? – S.N Feb 11 '21 at 20:04
  • You never needed the address of Zookeeper since you're using `--link` and that uses the `--name` given to the zookeeper container. – OneCricketeer Feb 11 '21 at 20:28
  • @OneCricketeer If I cannot use --link and --name for zookeeper, I should use its ip address . Is it right? – S.N Feb 11 '21 at 20:39
  • @TonyBrand Still no. https://docs.docker.com/network/network-tutorial-standalone/#use-user-defined-bridge-networks – OneCricketeer Feb 11 '21 at 21:07