3

I am starting a container with kafka via docker-compose successfully. By successfully I mean, I can log in such kafka container and produce and consume messages straight from there. Also I can see another container (filebeat) depending on such kafka container connecting successfully.

docker-compose.yml

version: '3.2'
services:

  zoo1:
    image: elevy/zookeeper:latest
    environment:
      MYID: 1
      SERVERS: zoo1
    ports:
      - "2181:2181"

  kafka1:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    depends_on:
      - zoo1
    links:
      - zoo1
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_HOURS: "168"
      KAFKA_LOG_RETENTION_BYTES: "100000000"
      KAFKA_ZOOKEEPER_CONNECT:  zoo1:2181
      KAFKA_CREATE_TOPICS: "log:1:1"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.5.2
    command: filebeat -e -strict.perms=false
    volumes:
      - "//c/Users/Cast/megalog-try-1/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
      - "//c/Users/Cast/megalog-try-1/sample-logs:/sample-logs"
    links:
      - kafka1
    depends_on:
      - kafka1

Nevertheless, when I try produce a very simple message from my local kafka to this container kafka I get connection refused.

I found someone with same exception but a bit similar scenario. I tried his solution without success. Basically it points to "adjust advertised.listeners to be the external IP". I did it in my docker-compose but nothing changed at all. And even if this had worked, I guess it would not be a final solution because for me since the guy is hardcoding his computer ip address in KAFKA_ADVERTISED_LISTENERS

So my question is how can I connect from my local kafka running in my Windows 10 to a kafka container in order to send a message?

Here are everything I have tried so far:

1 - trying with local ip

ipconfig
...
Endereço IPv4. . . . . . . .  . . . . . . . : 192.168.129.97

kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log

C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
>[2020-03-02 16:54:44,155] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-03-02 16:54:46,212] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

2 - trying with docker machine ip

C:\tools\kafka\bin\windows>docker-machine ip
192.168.99.100

kafka-console-producer.bat --broker-list 192.168.99.100:9092 --topic log


[2020-03-02 17:00:22,500] WARN [Producer clientId=console-producer] Error connecting to node kafka1:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka1
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
        at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
        at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
        at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:69)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1126)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1017)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:538)
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
        at java.base/java.lang.Thread.run(Thread.java:834)

3 - trying with container ip

docker ps
...
3eb114e24b53        wurstmeister/kafka                                    "start-kafka.sh"      0.0.0.0:9092->9092/tcp                                     megalog-try-1_kafka1_1

docker inspect 3eb114e24b53
...
"IPAddress": "172.18.0.4",

C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 172.18.0.4:9092 --topic log
>testing
[2020-03-02 16:57:54,471] WARN [Producer clientId=console-producer] Connection to node -1 (/172.18.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
org.apache.kafka.common.KafkaException: Producer closed while send in progress
        at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:888)

*** added after CRU's suggestion

C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_zoo1_1 bash
bash-4.3# ls
bin            entrypoint.sh  home           lib64          mnt            root           sbin           sys            usr            zookeeper
dev            etc            lib            media          proc           run            srv            tmp            var
bash-4.3# cd zookeeper/
bash-4.3# ls
bin                   conf                  data                  lib                   log                   wal                   zookeeper-3.4.10.jar
bash-4.3# cd conf
bash-4.3# ls
log4j.properties  zoo.cfg
bash-4.3# cat zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/wal
#snapCount=100000
autopurge.purgeInterval=1
clientPort=2181
quorumListenOnAllIPs=true
server.1=zoo1:2888:3888bash-4.3#

and from kafka container

C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# pwd
/
bash-4.4# ls
bin    dev    etc    home   kafka  lib    lib64  media  mnt    opt    proc   root   run    sbin   srv    sys    tmp    usr    var
bash-4.4# cd kafka
bash-4.4# ls
kafka-logs-0f385d1d435e
bash-4.4# cd kafka-logs-0f385d1d435e/
bash-4.4# ls
app_logs-0                        log-0                             meta.properties                   replication-offset-checkpoint
cleaner-offset-checkpoint         log-start-offset-checkpoint       recovery-point-offset-checkpoint  request_logs-0
bash-4.4# cat meta.properties
#
#Mon Mar 02 21:09:33 GMT 2020
cluster.id=-qitBVmjSUGo7Zd7P5cetw
version=0
broker.id=1
bash-4.4#

*** added

After stumbled in this article I could write a docker-compose to run kafka container that my local SpringBoot can send message. Nevertheless, I have to hard code the docker-machine ip manually in my docker-compose. Well, I don't think this is the only way otherwise everytime I start Docker ToolBox I will have to run docker-machine ip and replace the IP Address. I am wondering if there is a more friendly solution.

Bellow 192.168.99.100 is my docker-machine ip.

version: "3.2"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS:
        "INTERNAL://kafka:9090,\
         EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS:
        "INTERNAL://kafka:9090,\
         EXTERNAL://192.168.99.100:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
        "INTERNAL:PLAINTEXT,\
         EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Based on Cricket_007's suggestion, I read carefully another article but it drove me the conclusion that I must use DNS resolution but I can't setup an exclusive dns resolution in my Windows machine as far as I understand.

Jim C
  • 3,957
  • 25
  • 85
  • 162
  • 2
    Please check advertised.listeners in server.properties and Server list in zookeeper.properties in config directory. These files are used for connecting to brokers and zookeeper. – c0der512 Mar 02 '20 at 20:49
  • I can´t find zookeeper.properties in config directory in zookeeper Docker container. Are you talking about zookeeper.properties in my Windows? BTW, I added above all zookeeper config I have found in my docker container – Jim C Mar 02 '20 at 21:10
  • Also in kafka docker container I can't find any server.properties. I will add above – Jim C Mar 02 '20 at 21:17
  • 2
    Try to run following bash command './zookeeper-shell.sh localhost:9092 ls2 /brokers/ids' Check for the broker id in the output. This will tell you if any broker is not connected to cluster. – c0der512 Mar 02 '20 at 22:19
  • @CRU this is the output: Connecting to localhost:9092 KeeperErrorCode = ConnectionLoss for /brokers/ids . Does it give you some clue? As far as I can see, I will never be able to ping localhost:9092 from my local Windows to a kafka running in a Docker container – Jim C Mar 03 '20 at 19:50
  • I'm sorry for ambiguity. In my configuration, Zkhost and kafka is running on same instance. You can change the localhost to appropriate docker DNS. I would suggest you to link majority of zookeeper hosts + brokers in docker. Run a kafka instance in windows and in server.properties provide zookeeper detail of docker zookeeper. This is a way to expand your cluster. With this configuration, you will experience more latency for replicas running on windows. – c0der512 Mar 03 '20 at 20:01

2 Answers2

2

  1. KAFKA_ADVERTISED_PORT is deprecated. Just use the advertised listeners.

  2. Your advertised listeners are the Docker service name, not a resolvable address to any code running outside of a container on that same Docker network.

  3. You'll want to add in KAFKA_LISTENER_SECURITY_PROTOCOL_MAP

  4. Just use an existing, functional Docker Compose

Btw, links is a deprecated Compose option too

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Very clear your points #1 and #3. Regard my question "how can connect from my local kafka running in my Windows 10 to a kafka container" from the link shared by you, the only solution is by using a DNS service, right? In this article there are three llisteners proposed: 1 BOB (port 29092) for internal traffic on the Docker network, 2 FRED (port 9092) for traffic from the Docker host machine (localhost) and the only one that is related to my question 3 ALICE (port 29094) for traffic from outside, reaching the Docker host on the DNS name never-gonna-give-you-up. Is DNS the only solution? – Jim C Mar 03 '20 at 19:36
  • 1
    DNS ultimately resolves to IP addresses at the end of the day. If you want to save the extra network hop, then skip DNS and advertise an IP. Then rather than `UnknownHostException`, you'd end up with "no route to host" or `ConnectionRefused` for a bad IP – OneCricketeer Mar 03 '20 at 22:22
  • So, is that correct to say that the only way for my local Windows 10 reaches kafka container running in same laptop is by setting docker-machine ip in docker-compose KAFKA_ADVERTISED_LISTENERS as I pasted above (KAFKA_ADVERTISED_LISTENERS... EXTERNAL://192.168.99.100:9092)? If so, that is fine but if there is another alternative I would prefer otherwise I will have to edit docker-compose every time I restart docker-machine and a new ip is provided. – Jim C Mar 04 '20 at 15:01
  • Just realized the link I gave was German. Whoops. Anyways, If you have port forwarded 9092 from the VM to your host, you only need to advertise localhost or 127.0.0.1 ; if there is no port forward, then yes, advertise an ip that will work from that client machine. And I don't think restarting docker machine creates new IPs – OneCricketeer Mar 04 '20 at 15:17
  • I am pretty sure restart docker machine can create new docker machine ip. Actually, whenever you restart there is a warnning sayng exactly this. – Jim C Mar 04 '20 at 15:34
  • Well, unless I am missing some extra trick, replacing my docker-machin ip by 127.0.0.1 didn't work. Here is what happen when I try to reach it: C:\tools\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 1 --topic test [2020-03-04 12:41:52,775] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) – Jim C Mar 04 '20 at 15:44
  • At this moment, the only way to make kafka docker container reachable from my own machine is by hard-coding docker-machine ip in docker-compose. Well, maybe usefull comment maybe not: I am using Docker Tool Box with Oracle Virtual Box and I have noticed it behaves different from Dockerr for Descktop with HyperV. – Jim C Mar 04 '20 at 15:49
  • 1
    I rarely use Windows nowadays (at least for development). But I would also use HyperV w/ MiniKube directly if I needed to – OneCricketeer Mar 04 '20 at 20:26
  • Actually, I remenber I read from Docker community recommending to use Docker for Desktop with HyperV instead of Docker ToolBox with VirtualBox and I do remenber not facing such much connection issues when I was using Docker for Desktop. BTW, I must use OpenShift (minishift version) and OpenShift demands VirtualBox which conflicts with HyperV.Thanks for your insights – Jim C Mar 05 '20 at 13:27
  • 1
    Are you sure about that? https://docs.okd.io/latest/minishift/getting-started/setting-up-virtualization-environment.html#for-windows – OneCricketeer Mar 05 '20 at 14:56
0

This worked for me, but for a Kafka Docker container running locally. The link that @OneCricketeer provided above helped me to connect from the client (the application, the console producer, the console consumer) running outside Docker to Kafka running in a Docker container, locally.

    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

or, editing config/server.properties in the Docker container to read:

   listeners=PLAINTEXT://:9092
   advertised.listeners=PLAINTEXT://localhost:9092
Alex
  • 1,066
  • 2
  • 9
  • 15