0

I am trying to integrate Kafka with clickhouse. I want to send in some data from my REST API to kafka, and pass it to clickhouse which would be my DB. So far, I was able to get Kafka to working indivually and pass data from one end to other. But when I tried to use it in docker, I am having some issues. Well, I can see the data being consumed at the kafka end, using a kafka tool. But my clickhouse apparently is not getting connected. This is the error, I keep getting when I run my docker:

clickhouse-server_1  | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
clickhouse-server_1  | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down

I have also added the log file below:

2020.03.19 10:16:11.637331 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637658 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637854 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.678748 [ 74 ] {} <Error> void DB::DDLWorker::runMainThread(): Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. Addresses: 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
 (Connection loss), Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x102d352c in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f2d989 in /usr/bin/clickhouse
2. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) @ 0xdd300e4 in /usr/bin/clickhouse
3. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xdd3070e in /usr/bin/clickhouse
4. ? @ 0xdd638b2 in /usr/bin/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::__1::vector<Poco::Net::SocketAddress, std::__1::allocator<Poco::Net::SocketAddress> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Timespan, Poco::Timespan, Poco::Timespan) @ 0xdd6099b in /usr/bin/clickhouse
6. zkutil::ZooKeeper::init(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd3fd53 in /usr/bin/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd405bd in /usr/bin/clickhouse
8. DB::Context::getZooKeeper() const @ 0xce5fb97 in /usr/bin/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0xce84c53 in /usr/bin/clickhouse
10. DB::DDLWorker::runMainThread() @ 0xce8eae3 in /usr/bin/clickhouse
11. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() const @ 0xce96031 in /usr/bin/clickhouse
12. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8f50b07 in /usr/bin/clickhouse
13. ? @ 0x8f4f00f in /usr/bin/clickhouse
14. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
15. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
 (version 20.3.3.6 (official build))

Here are some of the configuration I am using:

CREATE TABLE IF NOT EXISTS filters (
  name String,
  value String
) ENGINE = Kafka SETTINGS
            kafka_broker_list = 'kafka://localhost:9092',
            kafka_topic_list = 'testtopic',
            kafka_group_name = 'test',
            kafka_format = 'JSONEachRow',
            kafka_num_consumers = 2

My kafka bootstrapping part:

producer = KafkaProducer(bootstrap_servers=['kafka://localhost:9092'], value_serializer=lambda m: json.dumps(m).encode('ascii'))

And my docker config part:

version: "3.6"

services:
  clickhouse-server:
    image: yandex/clickhouse-server
    volumes:
      - ./clickhouse/config.xml:/etc/clickhouse-server/config.xml
      - ./clickhouse/zookeeper-servers.xml:/etc/clickhouse-server/conf.d/zookeeper-servers.xml
      - ./shared/ch-data/clickhouse:/var/lib/clickhouse
    depends_on:
      - kafka
    links:
      - kafka
    ports:
      - 9000:9000

  clickhouse-client:
    image: yandex/clickhouse-client
    entrypoint:
      - /bin/sleep
    command:
      - infinity

  kafka:
    image: wurstmeister/kafka:2.11-1.0.2
    volumes:
      - ./shared/ch-data/kafka:/data
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: 192.168.1.86
    links:
     - zookeeper
    ports:
     - 9092:9092
     - 9094:9094

  zookeeper:
    image: zookeeper
    volumes:
      - ./shared/ch-data/zookeeper:/data
    ports:
      - 2181:2181

I am particularly confused on the bootstrap_Servers and KAFKA_ADVERTISED_LISTENERS part. So far as I know, I have provided my host IP as the KAFKA_ADVERTISED_LISTENER, from what I understood from here

Harshita
  • 57
  • 8

1 Answers1

1

Seems you have misunderstood Docker networking. Most importantly, links feature is deprecated, so I'm not sure where you copied your compose file from

localhost will always be the current container, not another one

Fix: kafka://localhost:9092 & bootstrap_servers=['kafka://localhost:9092'] to use the actual Docker service name

So far as I know, I have provided my host IP as the KAFKA_ADVERTISED_LISTENER

  1. You only need the host IP if you're using more than one physical machine. This isn't required for clickhouse in the same compose file, on the same machine
  2. You only need to advertise localhost if you are using Kafka clients outside of Docker, on the same physical machine (for example, your producer script)
  3. You should always advertise the kafka container service name for clients within the Docker network. In the file provided, this means you should set KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092, at the very least (I'm not sure what you're using port 9094 for, but that's a separate issue)

All in all, KAFKA_LISTENERS would have 3-4 ports with the fourth being for inter security protocol for replication. How many you advertise is up to you

And you'd add the protocol map, as discussed in that blog post.

Note that you're not required to use any particular kafka or Zookeeper docker image, and those compose files provided in the blog work correctly. Plus, never trust that two different Docker images share the same set of Environment variables.

The wurstmeister image has its own Github pages discussing the listeners

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245