1

Installed the standalone ksqlDB in my local computer.

While running the ksql server, i am getting the following error

Remote server at http://ksqldb-server:8088 does not appear to be a valid KSQL
server. Please ensure that the URL provided is for an active KSQL server.

The server responded with the following error: 
Error issuing GET to KSQL server. path:/info
Caused by: java.net.UnknownHostException: failed to resolve 'ksqldb-server'
    after 2 queries 
Caused by: failed to resolve 'ksqldb-server' after 2 queries

PS: my docker-compose.yml looks like as follows

---
version: '2'

services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.23.1
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: localhost:9092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.23.1
    container_name: ksqldb-cli
    depends_on:
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

While running docker-compose up, it is giving following error

 [2022-01-11 07:21:54,280] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
ksqldb-server    | org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata
ksqldb-server    | [2022-01-11 07:21:54,282] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:68)
ksqldb-server    | java.lang.RuntimeException: Failed to get Kafka cluster information
ksqldb-server    |  at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:107)
ksqldb-server    |  at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:669)
ksqldb-server    |  at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:154)
ksqldb-server    |  at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:61)
ksqldb-server    | Caused by: java.util.concurrent.TimeoutException
ksqldb-server    |  at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
ksqldb-server    |  at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
ksqldb-server    |  at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
ksqldb-server    |  at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:105)
ksqldb-server    |  ... 3 more
ksqldb-server exited with code 255
^CGracefully stopping... (press Ctrl+C again to force)
Stopping ksqldb-cli    ... done

in server.properties file, i added

listeners=PLAINTEXT://localhost:9092

and restarted kafka but it is not solved the problem.

kafka consumer and producer are working fine. the problem is only with kSQL

Any help is appreciated

Riyas
  • 299
  • 4
  • 15
  • Please show how you are running ksql and connecting to it. For example, why aren't you connecting to `http://localhost:8088`? – OneCricketeer Jan 10 '22 at 15:08
  • @OneCricketeer I didn't understand u r question properly. But added more details to my question – Riyas Jan 11 '22 at 07:40
  • `KSQL_BOOTSTRAP_SERVERS: localhost:9092` is not correct since this address refers to the KSQL container. KSQL requires a running Kafka cluster. Where is Kafka running? If Kafka is not a container, see https://stackoverflow.com/questions/68401788/connect-to-kafka-on-host-from-docker-ksqldb – OneCricketeer Jan 11 '22 at 16:03

1 Answers1

0

There are two things that are wrong in this config:

  1. You need to specify in the ksqldb-server yaml the name of the service which you have used to expose the kafka-cluster, if you used strimzi it usually ends up with "kafka-bootstrap"

  2. The second error is that you also need to expose the ksqldb-server so that the ksql-cli can connect to that using the ENV variable KSQL_BOOTSTRAP_SERVERS.

If you want there is a template ksql-db-service from the helm chart you can use:

apiVersion: v1
kind: Service
metadata:
  name: my-ksql-db-service
  namespace: default
spec:
  ports:
  - name: http
    port: 8088
    protocol: TCP
    targetPort: ksqldb-server
  selector:
    app.kubernetes.io/name: ksqldb-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

You need to do a similar thing for kafka if you haven't already but exposing port 9092.

However this process might be error-prone since you have to write manually a lot of yamls. There is no need to reinvent the wheel. Use helm charts and operators!

You can use this job template to execute the query:

apiVersion: batch/v1
kind: Job
metadata:
    namespace: default
    name: first-query
spec:
    template:
        spec:
            containers:
                - name: ksqldb-query
                  imagePullPolicy: IfNotPresent
                  image: confluentinc/cp-ksqldb-cli
                  command: ['/bin/bash']
                  args:
                      - -c
                      - >-
                          ksql --execute "select * from your_source emit changes" --config-file ksql-server.properties <KSQL_BOOTSTRAP_SERVERS>
                  envFrom:
                      - configMapRef:
                            name: my-ksql-db-4e876644-ksqldb
            restartPolicy: 'Never'

You can template better the yaml-job but did not want to complicate things

Let me know if something is not clear

Cr4zyTun4
  • 625
  • 7
  • 18