2

I'm trying to integrate toxy-proxy with a kafka cluster, but seems to not work.

The docker-compose file looks something like this:

kafka:
image: confluentinc/cp-kafka:latest
depends_on:
  - zookeeper
environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
  KAFKA_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

toxiproxy:
image: 'ghcr.io/shopify/toxiproxy:2.4.0'
ports:
  - 8474:8474
  - 29092:29092

Also my application is set up like this:

my_application:
image: image
environment:
  KAFKA_BROKERS_URL: toxiproxy:8474

The problem is that my application does not connect to kafka. Not sure how should be done. I used this https://github.com/Shopify/sarama/blob/main/docker-compose.yml as an example. Also I'd like to know how to call kafka from my localhost, so not from inside a docker container.

Basically this works now

kafka:
image: confluentinc/cp-kafka:latest
depends_on:
  - zookeeper
ports:
  - 29092:29092
environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1


  toxiproxy:
container_name: toxiproxy
image: 'ghcr.io/shopify/toxiproxy:latest'
command: ['-config', '/config/toxiproxy.json','-host', '0.0.0.0']
volumes:
  - ./config/toxiproxy.json:/config/toxiproxy.json
ports:
  - target: 8474
    published: 8474
    protocol: tcp
    mode: host
depends_on:
  - kafka

Where the config file looks like this:

[
{
    "name": "kafka_proxy",
    "listen": "[::]:45390",
    "upstream": "kafka:9092",
    "enabled": true
}

]

However because after the application makes a first connection to Kafka it will ask for cluster configuration, and will be served hostnames according to ADVERTISED_LISTENERS, which will be set to the actual kafka container and not to my proxy.

This I still don't know how to do, to set the advertised_listeners to use the proxy as well. I've tried setting it like this, but kafka stops working.

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://toxiproxy:45390
BadChanneler
  • 1,450
  • 1
  • 12
  • 20
  • Why exactly do you need a proxy? What is port 8474 supposed to do differently from 29092 when only the Kafka server opens port 29092, not the proxy? Or if you do need one, why not one built for Kafka? https://github.com/grepplabs/kafka-proxy – OneCricketeer Sep 14 '22 at 12:55
  • _how to call kafka from my localhost_ - https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker#51634499 – OneCricketeer Sep 14 '22 at 12:58
  • 2
    Toxiproxy is a framework for simulating network conditions and I want to simulate a failing communication between my app and kafka. I use it in my automation tests. – BadChanneler Sep 14 '22 at 13:28
  • Okay, fair, so in that case, how does 29092 port for the proxy container know anything about Kafka? Seems like you are missing some other configs – OneCricketeer Sep 14 '22 at 15:42
  • yeah, I think you're right, but I don't know how and didn't find any documentation – BadChanneler Sep 15 '22 at 07:59
  • Based on what I see, you should be adding proxies via http requests on port 8474 or overriding the CMD for the container to do the same, or maybe mounting some config file, or use toxiproxy-cli – OneCricketeer Sep 15 '22 at 14:27
  • I've updated my question and found out how to do it, but I still don't know how to use toxiproxy as an KAFKA_ADVERTISED_LISTENERS... – BadChanneler Sep 16 '22 at 06:42
  • Where are you trying to connect a kafka client from? If it's your host machine, you need to advertise `localhost:45390`. Then you need your compose file to also forward that port for the proxy, and you shouldn't need host mode. You can use `kcat -L` to test listeners – OneCricketeer Sep 16 '22 at 13:52

0 Answers0