0

I want to run kafka in a single node, single broker, in one of computers on our network and be able to access it from other machines. for example by running docker-compose on 192.168.0.36 I want to access it from 192.168.0.19. since we can't use any Linux distribution I have to run kafka as a docker container on windows. I know there are already a ton of questions and documents on this topic including this question and this example and also this blog post, but unfortunately none of them worked out for me.

this is the compose file I'm using right now:

version: '3.7'
services:

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
      - "2181:2181"
    expose:
      - "2181"
    volumes:
    - type: bind
      source: "G:\\path\\to\\zookeeper"
      target: /opt/zookeeper-3.4.6/data
    

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    expose:
      - "9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9093, OUTSIDE://192.168.0.36:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT, OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
      KAFKA_LOG_DIRS: "/kafka"
 
    volumes:
    - type: bind
      source: "G:\\path\\to\\logs"
      target: /kafka/

things I tried for debugging the issue:

  • alraedy tried all the different configurations in mentioned questions and blog posts.

  • I can access Kafka from 192.168.0.36 which is machine running docker-compose but not from 192.168.0.19 (NoBrokersAvailable error in kafka-python).

  • just to see if it's internal networking problem or not, I tried a similar docker-compose file running a falcon API using gunicorn and I can call the API from 192.168.0.19.

  • I also tried the windows telnet tool to see the 9092 port is accessible from different machines, it's accessible from 0.36 but not from 0.19.

  • tried using a custom network like this one

I'm testing the connection using python's kafka-python package. we have a multi-broker kafka running on our on-premise kubernetes cluster and it's working fine, so I don't think my testing scripts have any issues.

UPDATE

as OneCricketeer suggested, I tried this solution with different configurations like 0.0.0.0:9092=>127.0.0.1:9092 and 192.168.0.36:9092=>127.0.0.1:9092. also disabled firewall. still getting NoBrokersAvailable but at least I can access 0.36:9092 from other machine's telnet now.

aSaffary
  • 793
  • 9
  • 22
  • Please show examples/errors of using kafkacat as the solution in the other question shows – OneCricketeer Aug 25 '21 at 06:01
  • I assume `KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092` is a typo? You cannot advertise the same port twice.... Also, are you running Docker in WSL2? Because that requires its own port forwarding and firewall rules (why not just run Kafka in WSL2 directly, if that's the case?) – OneCricketeer Aug 25 '21 at 06:05
  • @OneCricketeer I will have to try and run kafka in wsl 2 if this solution doesn't work, but since we want to have this alternative for quick testing on external systems we prefer being able to run it via docker-compose. – aSaffary Aug 25 '21 at 11:31
  • @OneCricketeer `kafkacat -b 192.168.0.36 -L` returns `ERROR: Failed to acquire metadata: Local: Broker transport failure.` and yes that was a typo i'll fix it now thanks. – aSaffary Aug 25 '21 at 11:39
  • Docker is in WSL2, though? Via Docker Desktop? Not using Docker Machine / VirtualBox? If so, then see similar issue/solution here https://github.com/microsoft/WSL/issues/5439#issuecomment-646855595 – OneCricketeer Aug 25 '21 at 12:18
  • @OneCricketeer yes I'm using WSL2. as I said I have no problem accessing kafka from localhost 192.168.0.36. but I tried your solution as well (which I'll update the question with it now) but still can't access kafka from other systems on network. – aSaffary Aug 26 '21 at 05:03
  • What were the exact `netsh interface portproxy add v4tov4` commands ran? – OneCricketeer Aug 26 '21 at 16:01
  • 1
    @OneCricketeer first tried `netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=127.0.0.1` then after it didn't work i tired `netsh interface portproxy add v4tov4 listenport=9092 listenaddress=192.168.0.36 connectport=9092 connectaddress=127.0.0.1` the second one disrupted my localhost access and i had to remove it. – aSaffary Aug 26 '21 at 16:10
  • 1
    So telnet works now? What about `kafkacat -L`? – OneCricketeer Aug 26 '21 at 16:20
  • 1
    @OneCricketeer so I removed all the netsh interfaces and its working now. thank you so much for your time. I just had to disable firewall (without adding interface proxies) such a silly mistake I'm sorry it's the first time I'm using windows for this kind of works so I made a rookie mistake. please post this suggestion as answer so I can accept it as and save others from making the same mistake. – aSaffary Aug 28 '21 at 04:17

0 Answers0