0

After reading these threads

. Kafka topic creation: Timed out waiting for a node assignment . Error while executing Kafka topic command: Timed out waiting for a node assignment. OS Win 10 . Docker kafka on MacOS M1 Issues stuck on configuring

I decided to go with @igor-fedorov guideline at the last thread

I've just followed it at my brand new MacBook Air M1 ... But I am still getting the damn [2021-10-22 22:37:07,031] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

So, my data:

  1. I cloned Apache/Kafka and checked out it to 2.4 branch (it's a project requirement)
  2. I successfully built it with ./gradlew
  3. I create the folder structure as you commented
  4. Docker file (yours but only exposing 9092 port)
  5. Server.properties
broker.id=0
listeners=PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092
advertised.listeners=PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092
zookeeper.connect=zookeeper:2181
zookeeper.connection.timeout.ms=6000
  1. docker-compose.yml. As you can see I aggregated schema-registry docker image too and using a common network definition. And I also didn't get if the server.properties conf prevail or do not prevail over docker-compose properties
version: '2'
services:
  zookeeper:
    #image: confluentinc/cp-zookeeper:5.4.2
    image: zookeeper:3.7.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ~/Codigo/ECC/kafka-build-local/volumes/zookeper/data:/data
      - ~/Codigo/ECC/kafka-build-local/volumes/zookeper/datalog:/datalog
    networks:
      - firefly

  kafka1:
    #image: confluentinc/cp-enterprise-kafka:5.4.2
    #image: fogsyio/kafka:arm64v8-2.2.0
    hostname: kafka1
    container_name: kafka1
    build: ./kafka_m1
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      #- "127.0.0.1:9092:9092"
    environment:
      #OJO: CON build PUEDE QUE EL FICHERO DE CONFIGURACIÓN PREVALEZCA SOBRE ESTA CONFIGURACIÓN?
      # kafka/config/server.properties
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092
      # Para local no es necesario métricas
      #KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka1:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      #CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_METRICS_ENABLE: "false"
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
    volumes:
      - ~/Codigo/ECC/kafka-build-local/volumes/kafka1:/var/lib/kafka/data
    networks:
      - firefly

  schema-registry:
    #image: confluentinc/cp-schema-registry:5.4.2
    image: eugenetea/schema-registry-arm64:latest
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - zookeeper
      - kafka1
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'
    volumes:
      - ~/Codigo/ECC/kafka-build-local/volumes/schema-registry:/data
    networks:
      - firefly


networks:
  firefly:
    driver: bridge
  1. ZooKeeper start up logs
2021-10-23 09:35:19,584 [myid:1] - INFO  [main:NIOServerCnxnFactory@660] - binding to port /0.0.0.0:2181
2021-10-23 09:35:19,597 [myid:1] - INFO  [main:WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2021-10-23 09:35:19,597 [myid:1] - INFO  [main:WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2021-10-23 09:35:19,598 [myid:1] - INFO  [main:ZKDatabase@133] - zookeeper.snapshotSizeFactor = 0.33
2021-10-23 09:35:19,598 [myid:1] - INFO  [main:ZKDatabase@153] - zookeeper.commitLogCount=500
2021-10-23 09:35:19,601 [myid:1] - INFO  [main:SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED
2021-10-23 09:35:19,633 [myid:1] - INFO  [main:FileSnap@85] - Reading snapshot /data/version-2/snapshot.17e
2021-10-23 09:35:19,640 [myid:1] - INFO  [main:DataTree@1709] - The digest in the snapshot has digest version of 2, , with zxid as 0x17e, and digest value as 93697964796
2021-10-23 09:35:19,670 [myid:1] - INFO  [main:ZKAuditProvider@42] - ZooKeeper audit is disabled.
2021-10-23 09:35:19,671 [myid:1] - INFO  [main:FileTxnSnapLog@372] - 55 txns loaded in 16 ms
2021-10-23 09:35:19,671 [myid:1] - INFO  [main:ZKDatabase@290] - Snapshot loaded in 73 ms, highest zxid is 0x1b5, digest is 88637075219
2021-10-23 09:35:19,677 [myid:1] - INFO  [main:FileTxnSnapLog@479] - Snapshotting: 0x1b5 to /data/version-2/snapshot.1b5
2021-10-23 09:35:19,683 [myid:1] - INFO  [main:ZooKeeperServer@543] - Snapshot taken in 6 ms
2021-10-23 09:35:19,693 [myid:1] - INFO  [main:RequestThrottler@75] - zookeeper.request_throttler.shutdownTimeout = 10000
2021-10-23 09:35:19,693 [myid:1] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@137] - PrepRequestProcessor (sid:0) started, reconfigEnabled=false
2021-10-23 09:35:19,711 [myid:1] - INFO  [main:ContainerManager@84] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0
2021-10-23 09:35:19,924 [myid:1] - INFO  [SyncThread:0:FileTxnLog@284] - Creating new log file: log.1b6
  1. Kafka start up logs
kafka1             | [2021-10-23 09:35:20,425] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
kafka1             | [2021-10-23 09:35:20,438] INFO Stat of the created znode at /brokers/ids/0 is: 452,452,1634981720431,1634981720431,1,0,0,72057717675327488,247,0,452
kafka1             |  (kafka.zk.KafkaZkClient)
kafka1             | [2021-10-23 09:35:20,438] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(kafka1,29092,ListenerName(PLAINTEXT),PLAINTEXT), EndPoint(localhost,9092,ListenerName(PLAINTEXT_HOST),PLAINTEXT)), czxid (broker epoch): 452 (kafka.zk.KafkaZkClient)
kafka1             | [2021-10-23 09:35:20,473] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
kafka1             | [2021-10-23 09:35:20,474] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
kafka1             | [2021-10-23 09:35:20,480] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
kafka1             | [2021-10-23 09:35:20,497] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
kafka1             | [2021-10-23 09:35:20,498] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
kafka1             | [2021-10-23 09:35:20,501] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 3 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
kafka1             | [2021-10-23 09:35:20,525] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:10000,blockEndProducerId:10999) by writing to Zk with path version 11 (kafka.coordinator.transaction.ProducerIdManager)
kafka1             | [2021-10-23 09:35:20,542] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
kafka1             | [2021-10-23 09:35:20,547] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
kafka1             | [2021-10-23 09:35:20,548] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
kafka1             | [2021-10-23 09:35:20,580] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
kafka1             | [2021-10-23 09:35:20,605] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka1             | [2021-10-23 09:35:20,632] INFO [SocketServer brokerId=0] Started data-plane processors for 2 acceptors (kafka.network.SocketServer)
kafka1             | [2021-10-23 09:35:20,636] INFO Kafka version: 2.4.2-SNAPSHOT (org.apache.kafka.common.utils.AppInfoParser)
kafka1             | [2021-10-23 09:35:20,636] INFO Kafka commitId: d0488d9a80ec55c4 (org.apache.kafka.common.utils.AppInfoParser)
kafka1             | [2021-10-23 09:35:20,636] INFO Kafka startTimeMs: 1634981720632 (org.apache.kafka.common.utils.AppInfoParser)
kafka1             | [2021-10-23 09:35:20,637] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
kafka1             | [2021-10-23 09:35:20,663] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(test-ton-build2-1, test-ton-build2-0, test-ton-build-0, _schemas-0, test-ton-build-1, test-ton-build-2, test-ton-build2-2) (kafka.server.ReplicaFetcherManager)
kafka1             | [2021-10-23 09:35:20,704] INFO [Log partition=test-ton-build-2, dir=/tmp/kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
kafka1             | [2021-10-23 09:35:20,707] INFO [Log partition=test-ton-build-2, dir=/tmp/kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 21 ms (kafka.log.Log)
kafka1             | [2021-10-23 09:35:20,710] INFO Created log for partition test-ton-build-2 in /tmp/kafka-logs/test-ton-build-2 with properties {compression.type -> producer, min.insync.replicas -> 1, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, retention.ms -> 604800000, segment.bytes -> 1073741824, flush.messages -> 9223372036854775807, message.format.version -> 2.4-IV1, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager)

And It continues checking just-created topics and ends with these lines

[2021-10-23 09:35:20,759] INFO [Partition test-ton-build2-1 broker=0] No checkpointed highwatermark is found for partition test-ton-build2-1 (kafka.cluster.Partition)
[2021-10-23 09:35:20,759] INFO [Partition test-ton-build2-1 broker=0] Log loaded for partition test-ton-build2-1 with initial high watermark 0 (kafka.cluster.Partition)
[2021-10-23 09:35:20,759] INFO [Partition test-ton-build2-1 broker=0] test-ton-build2-1 starts at leader epoch 5 from offset 0 with high watermark 0. Previous leader epoch was -1. (kafka.cluster.Partition)

So, When I log into Kafka docker and I perform the create topic command

sh-4.4# kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test-ton-build4

The topic is created

sh-4.4# kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test-ton-build4
sh-4.4# kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
_schemas
test-ton-build
test-ton-build2
test-ton-build4
sh-4.4#

but when I perform the same command from my Mac, there is the Timed out waiting for a node assignment. error

agil.coto in ECC kafka-build-local> kafka_m1/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test-ton-build5
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2021-10-23 13:00:36,215] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
    at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:225)
    at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:194)
    at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:189)
    at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:217)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
 (kafka.admin.TopicCommand$)

any one could help me?

Ton
  • 101
  • 1
  • 5
  • 1) `KAFKA_LISTENERS` value needs to just be `PLAINTEXT://0.0.0.0:9092`Try to only use the environment variables to set this, not a property file 2) You need to verify those variables actually exist for **that docker image**. For example, you mounted `/var/lib/kafka/data` but your logs indicate that is not the storage directory. Therefore this image you're using is not having the exact same config as the Confluent image – OneCricketeer Oct 23 '21 at 13:20
  • As I stated before, there is no confluent Kafka image, it's a local build off apache/kafka GitHub account. Since it is a docker compose using "build" instead of "image", I have to use server.properties (At least I do not know how to use env variables overriding properties). If I use your "0.0.0.0" solution it works but then "schema-registry" fails on startup. In fact I am using your configuration at the thread you told me but swapping 9092 and 29092 ports. On the other hand I do not get your point regarding `/var/lib/kafka/data` that folder does exist at docker machine. – Ton Oct 25 '21 at 06:18
  • By the way, that port configuration is working for my linux mates at the project – Ton Oct 25 '21 at 07:17
  • The image doesn't really matter. Your listeners are incorrect. And you cannot just copy environment variables between any set of Kafka docker images. The schema registry needs to use Kafka as the "Kafka store connection", not Zookeeper. And again, the environment variables might need to be different in order to set that property – OneCricketeer Oct 25 '21 at 13:11
  • My listeners: `advertised.listeners=PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092` while your listeners `PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092` I just swap the port. My command from Mac host `--bootstrap-server localhost:9092` . Your command conf `outside the Docker network (including CLI tools you might have installed locally), use localhost:29092`. So again I am swapping the port, accordingly to my conf. Both "listeners" and "advertised.listeners" have the same value. What exactly is wrong? – Ton Oct 25 '21 at 14:43
  • I already got the env variables at my `kafka1` machine, the `server.properties` is configured the same way as env variables, the ports are open (`curl telnet://kafka1:29092` and `curl telnet://localhost:9092`get connected and connection is not closed) the `kafka1` docker machine has port forwarding `0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka1` and still it is not working ... the `curl telnet://localhost:9092` from my Mac m1 gets connected but the connection is closed immediately `* Connected to localhost (::1) port 9092 (#0) * Closing connection 0` – Ton Oct 26 '21 at 05:48
  • 1
    I FIXED IT! . Confirmed: "environment" section at docker-compose.yml is useless IF using "build" section, It is built from `kafka/config/server.properties`. Docker-compose is just `build: ./kafka_m1 depends_on: - zookeeper ports: - "9092:9092" volumes: - :<"logs.dir" property> networks: - `. So `server.properties` properties: `advertised.listeners=PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092` IF getting to kafka from your Mac M1 at standard 9092 port and the TRICK ... `listeners=PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://0.0.0.0:9092` – Ton Oct 26 '21 at 10:24
  • Environment section is not useless if using the build property. The environment variables only get used if your image actually uses them, like I was saying. Building Kafka with Gradle and running it directly doesn't do that – OneCricketeer Oct 26 '21 at 13:30
  • 1
    Yes, of course. I can denied it. But as I said before, I just downloaded kafka repo and build it. Going this way is the `server.properties`which rules. Thank you very much for your insights, by the way, I was able to fix my set up with your answer at the referenced thread – Ton Oct 27 '21 at 14:41

0 Answers0