0

For now, I set my producer on machine A, and consumer on machine B, both are docker container, and both are under same WIFI connection. How to set up?

Now I can set it work on each side. The following is the docker-compose.yaml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper

same on both machine.

I would like to know how to set the hostname and IP address? I saw someone said that docker overlay network, I am not so sure if that is suitable in this scenario. Actully I have already tried

docker swarm init

on one machine, and let another join, but it cannot join, not sure how to use this. Is there anyone has some experience on this issue? Please share some comments with me!

Edward Chang
  • 141
  • 1
  • 2
  • 11
  • 1) KAFKA_ADVERTISED_HOST_NAME is a deprecated property, uses only listeners 2) You need to configure the advertised listeners with the external IP/hostname of the machine **and** setup a port forward (Compose ports property) – OneCricketeer Jun 05 '21 at 21:51
  • Ultimately, I'd suggest using https://strimzi.io on a k3s cluster rather than doing this yourself with Swarm unless you really want to struggle with network setup before you have a functional containerized Kafka cluster – OneCricketeer Jun 05 '21 at 21:52
  • @OneCricketeer Thanks for your reply. For both under same WIFI network, how to set the IP properly, since under the same WIFI would get the same public IPv4 address, not sure how to set it? – Edward Chang Jun 06 '21 at 10:42
  • If you're in the same LAN network, use the IP given by your local router. For two different machines, that would not be the same. If you are asking to connect from a machine outside of your local network, you need to setup multiple port forwards in the router. Kafka is not unique in that aspect except that you need to then setup the advertised listeners property such that it uses the forwarded port number and your ISP provided external IP – OneCricketeer Jun 06 '21 at 12:12
  • @OneCricketeer, thanks for your comments. Where can I check the ip given by my local router. I just checked on my ubuntu machine "wired setting ", there are "IPv4 address" and "Default Route", but I tried to ping these two address on another ubuntu machine, didn't get any response. Or is that address should be some others? like from ifconfig? Or do I need to setting up something to make it can be ping from another machine? – Edward Chang Jun 06 '21 at 13:47
  • Machine A (want to set it as kafka consumer) with "ens33" "IPv4 address" 192.168.a.b, Machine B (want to set it as kafka producer) with "wlan0" "IPv4 address" 172.20.10.3, Both are docker container, and both docker0 are the same "172.17.0.1", How do I set the following 2 parameters: KAFKA_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_ADVERTISED_HOST_NAME: localhost – Edward Chang Jun 06 '21 at 14:17
  • The Machine A is a VM on my laptop, Machine B is a NVIDIA NX (also ubuntu) – Edward Chang Jun 06 '21 at 20:28
  • I suggest not using Docker here, first of all, if you're already using Ubuntu. And this really isn't enough room here to explain how to port forward your VMs. Follow the linked duplicate to get it working on one machine first. As written there, `KAFKA_LISTENERS: PLAINTEXT://localhost:9092` is never correct for this scenario. To get IPs, use `ifconfig`. wlan0 IP on the host is what you need to put into the advertised listeners variable and the advertised port needs to match the port used by the listeners. – OneCricketeer Jun 07 '21 at 13:15
  • Ping also isn't adequate to check things. Use netcat for checking ports as well, and also see the `kafkacat` tool usage in the blog of the linked question – OneCricketeer Jun 07 '21 at 13:15

0 Answers0