2

As the title states I am having issues getting my docker client to connect to the docker broker

Errors:

%3|1647161877.851|FAIL|fc24c271e73f#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2022-03-13T08:57:57.851241551Z %3|1647161877.851|ERROR|fc24c271e73f#producer-1| [thrd:app]: fc24c271e73f#producer-1: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

Server setup: Community docker-compose.yml from step 1 of guide:

https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html

Client setup:

.NET Confluent.Kafka producer

Procedures so far:
I initially thought it would be an issue with localhost in the docker container and using docker option:

--net=host

also fixed the issue. However this has the side effect of removing published ports and is no good.

I then tried using

--add-host host.docker.internal:host-gateway

which I found in another post (cannot remember which) however this does not work. I initially thought there was an issue with it since the --net=host works but when I tried it with a SignalR sample project it worked straight away.

I have come across multiple posts suggesting changes to

KAFKA_ADVERTISED_LISTENERS

However I find it odd that the host forwarding is not working and changes to this would make a difference. I have nonethelss tried out multiple combinations of it whether it be the broker ip of localhost to 0.0.0.0 etc. (stuff I found across different posts) but with no success.

Trying to get a better understanding of what is going on I came across this post:

https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

Which explains the 2 step process involved in connecting to a broker. Looking at the errors I received with the /bootstrap it seems to already fail on the initial connection. This however should be an indication that the immediate issue is not with with KAFKA_ADVERTISED_LISTENERS however I may be wrong in that assumption.

The post does have a docker to docker scenario but that is being done using custom network bridge which I do not want to have to use for this.

Anybody has any ideas of what I should do? Perhaps knows the exact changes to make and where.

dev01
  • 97
  • 1
  • 1
  • 9

1 Answers1

3

It appears your code is trying to connect to port 9092.

PLAINTEXT://broker:29092 is the listener you want to connect to, assuming your own code is in a container, as well.

If it isn't, then you can use localhost:9092 without host networking, but you need to have these variables

KAFKA_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://0.0.0.0:9092 
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092

done using custom network bridge

Compose defaults to create its own network bridge unless you define one separately and link all services to it.

not want to have to use for this.

You'll need a bridge unless you're running the code on a Linux host where --net=host will actually work. Then you should be able to use localhost:9092.

You shouldn't need host networking, though, if you understand how the listeners work

has the side effect of removing published ports

Not exactly. The exposed ports of the container should automatically be exposed on the host

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks for reply. I had already tried many combinations between code,docker and listeners. For the above it was simply connecting to localhost:9092. The KAFKA_ADVERTISED_LISTENERS is already set like this in the compose file. Adding KAFKA_LISTENERS causes the broker(cp-server) to no longer start up. Is --add-host not an option? In which case I would have to use custom network bridge (client is not part of compose). "has the side effect of removing published ports" I wrote this incorrectly. I meant to say the mapped ports – dev01 Mar 23 '22 at 21:12
  • Adding `KAFKA_LISTENERS` shouldn't prevent the broker from starting. You'll need to show the logs that include any error that happened when you added that. And no, `--add-host` is not the correct solution to the problem you're having; using a bridge network is. Why not add to the same compose file? What is wrong with using `--network` flag? – OneCricketeer Mar 23 '22 at 22:13
  • I got it working and you are right. Using --network makes more sense. I have different solutions each with separate projects that can communicate with Kafka so keeping it as a separate docker-compose makes more sense. This way I can up it on its own and do the projects separately. – dev01 Apr 23 '22 at 06:23