Connect to Kafka does not work when called by the openfaas service

Question

Connect to Kafka normally via Java without using openfaas. (Successful)

Connect to Kafka does not work when called by the openfaas service.

When running a function as service created by openfaas, it cannot connect to Kafka. (Kafka is running on Docker)

docker-compose.yml :

version: '2'
services:
  zookeeper:
    container_name: zookeeper
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181  

kafka:
    container_name: kafka
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1 
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:29092      
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Error:
Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.

String topic = "Request";
String KAFKA_BROKERS = "localhost:9092";

Properties props = new Properties();
            props.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA_BROKERS);
            props.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
            props.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class.getName());

producer = new KafkaProducer(props);

Reminder that OpenFaaS starts containers. Localhost is the app container, not a Kafka broker. Please edit your question to clarify where Kafka **is running** — OneCricketeer, Feb 03 '22 at 15:44
That's not enough information. Where is the Java code running? If that is also in Docker, then `localhost` refers to the Java container, not the container. See https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker — OneCricketeer, Feb 07 '22 at 15:31
Java also runs on Docker. In fact, openFaaS Service and Kafka both run on Docker — Hossein M, Feb 07 '22 at 15:53
@OneCricketeer I used kafka: 9092 on the Java side inside Docker but the connection is not established — Hossein M, Feb 08 '22 at 07:27

OneCricketeer · Answer 1 · 2022-02-08T15:15:19.813

0

Java also runs on Docker

localhost refers to that Java container, which is not a Kafka service.

Make sure your containers run in the same Docker network, then you need to reach external services using their container names, e.g. "kafka:9092", assuming that is what the container hostname is.

You will also need to make sure KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092

edited Feb 08 '22 at 15:15

answered Feb 07 '22 at 15:56

OneCricketeer

179,855
19
132
245

It is not working – Hossein M Feb 08 '22 at 07:50
Like I said, the containers need to be on the same network One way to do this is to actually put your Java service in the same Compose file. Otherwise, you must manually set the network value when you run the Java container. Then test its connection... For example, `docker exec` into the Java container, then try to `ping` the other. Still, `localhost` isn't correct, and I can't really debug your network setup for you – OneCricketeer Feb 08 '22 at 15:14
The main problem is that the Java service is build and deploy by faas-cli.exe and placed in Docker. How can I tell faas-cli.exe with what IP to put it in Docker? – Hossein M Feb 09 '22 at 09:36
I said nothing about ip addresses. All I said was to exec and ping using _service names_. E.g. The values you'd see with `docker ps`. Your basic issue here is that Docker Compose creates its own network. No container outside that compose file/network will be able to reach those services unless you manually use `docker network` commands – OneCricketeer Feb 09 '22 at 15:53

Connect to Kafka does not work when called by the openfaas service

1 Answers1