2

I'm running ksqldb-server from a docker-compor found here https://ksqldb.io/quickstart.html#quickstart-content

My kafka bootstrap server is running on the same VM in standard alone mode. I can see the messages in one topic with a console consumer:

sudo kafka-avro-console-consumer --from-beginning --bootstrap-server localhost:9092 --topic source-air-input  --property print.key=true --max-messages 2

Unfortunatly running ksql from docker gives me this error.

ksqldb-server    | [2021-07-15 23:12:58,772] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:66)
ksqldb-server    | java.lang.RuntimeException: Failed to get Kafka cluster information
ksqldb-server    |      at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:107)
ksqldb-server    |      at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:624)
ksqldb-server    |      at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:152)
ksqldb-server    |      at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
ksqldb-server    | Caused by: java.util.concurrent.TimeoutException
ksqldb-server    |      at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
ksqldb-server    |      at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
ksqldb-server    |      at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:105)

My docker-compose.yml is the following.

 ---
version: '3.9'

services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.18.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: host.docker.internal:9092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.18.0
    container_name: ksqldb-cli
    depends_on:
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

I tried many possible configurations for the address without success. What might be wrong? I tried the suggestions from this question From inside of a Docker container, how do I connect to the localhost of the machine? without success.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Thadeu Melo
  • 947
  • 14
  • 40
  • 1
    Kafka is not running inside the KSQL container, so the bootstrap cannot be localhost – OneCricketeer Jul 15 '21 at 23:27
  • @OneCricketeer I thought about that, so, what is the address I shoud give? – Thadeu Melo Jul 15 '21 at 23:44
  • @OneCricketeer it is not a duplicated post. I tried everything there. – Thadeu Melo Jul 16 '21 at 16:23
  • So, what error do you get with `KSQL_BOOTSTRAP_SERVERS: docker.host.internal:9092`? Note: You **also**, need to modify `advertised.listemers` to list the same. – OneCricketeer Jul 16 '21 at 16:40
  • Secondly: You don't really need Docker - you can install KSQL as an RPM https://ksqldb.io/quickstart-standalone-rpm.html#quickstart-content – OneCricketeer Jul 16 '21 at 16:43
  • 1
    @OneCricketeer I got this error, org.apache.kafka.common.config.ConfigException: Invalid value docker.host.internal:8088 for configuration listeners: Not valid URL: unknown protocol: docker.host.internal – Thadeu Melo Jul 16 '21 at 16:58
  • 1
    @OneCricketeer running standalone I got this error: Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: – Thadeu Melo Jul 16 '21 at 17:00
  • 8088 is the KSQL REST endpoint, not the Kafka broker – OneCricketeer Jul 16 '21 at 17:01
  • No solution again: Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers – Thadeu Melo Jul 16 '21 at 17:03

1 Answers1

2

Modify Kafka's server.properties

listener.security.protocol.map=PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_LOCAL:PLAINTEXT

listeners=PLAINTEXT_DOCKER://:29092,PLAINTEXT_LOCAL://localhost:9092

advertised.listeners=PLAINTEXT_DOCKER://host.docker.internal:29092,PLAINTEXT_LOCAL://localhost:9092

inter.broker.listener.name=PLAINTEXT_LOCAL

Update your Compose like so to point at the host rather than itself

version: '3.9'

services:

  # TODO: add schema-registry
  #   environment: 
  #     SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: host.docker.internal:29092
  #   extra_hosts:
  #     - "host.docker.internal:host-gateway"

  # or any other Kafka client 
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.18.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
       KSQL_BOOTSTRAP_SERVERS: host.docker.internal:29092
       ...

(Tested on Mac), Getting /info endpoint of KSQL

http :8088/info
HTTP/1.1 200 OK
content-length: 133
content-type: application/json

{
    "KsqlServerInfo": {
        "kafkaClusterId": "ZH2-h1W_SaivCW0qa8DQGA",
        "ksqlServiceId": "default_",
        "serverStatus": "RUNNING",
        "version": "0.18.0"
    }
}

Replace all host.docker.internal above with the external hostname/IP of the machine, if Kafka is a remote server

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245