I am using Telegraf with Kafka consumer input plugin to forward the messages to InfluxDB. The Telegraf logs show the following when trying to consume from Kafka server
Apr 12 11:25:13 algtigtelegraf telegraf[848255]: 2021-04-12T08:25:13Z E! [inputs.kafka_consumer] Error in plugin: kafka: error while consuming telegraf/0: read tcp TELEGRAF_IP:38494->KAFKA_IP:9093: i/o timeout
Apr 12 11:25:28 algtigtelegraf telegraf[848255]: 2021-04-12T08:25:28Z E! [inputs.kafka_consumer] Error in plugin: kafka: error while consuming telegraf/0: kafka server: The provided member is not known in the current generation.
Apr 12 11:25:28 algtigtelegraf telegraf[848255]: 2021-04-12T08:25:28Z E! [inputs.kafka_consumer] Error in plugin: kafka: error while consuming telegraf/0: kafka server: The provided member is not known in the current generation.
Apr 12 11:25:28 algtigtelegraf telegraf[848255]: 2021-04-12T08:25:28Z E! [inputs.kafka_consumer] Error in plugin: kafka: error while consuming telegraf/0: kafka server: The provided member is not known in the current generation.
Apr 12 11:25:28 algtigtelegraf telegraf[848255]: 2021-04-12T08:25:28Z E! [inputs.kafka_consumer] Error in plugin: kafka: error while consuming telegraf/0: kafka server: The provided member is not known in the current generation.
By looking at the Kafka logs, I can see the following related to the above error:
[2021-04-12 08:26:42,559] INFO [GroupCoordinator 0]: Preparing to rebalance group telegraf_metrics_consumers in state PreparingRebalance with old generation 238 (__consumer_offsets-6) (reason: removing member Telegraf-bd5ba2ac-e037-4d06-91d3-155d9cc63981 on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:26:42,560] INFO [GroupCoordinator 0]: Group telegraf_metrics_consumers with generation 239 is now empty (__consumer_offsets-6) (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:01,424] INFO [GroupCoordinator 0]: Dynamic Member with unknown member id joins group telegraf_metrics_consumers in Empty state. Created a new member id Telegraf-b1168722-a4f9-42cc-b0c3-10c78e2cf4d1 for this member and add to the group. (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:01,424] INFO [GroupCoordinator 0]: Preparing to rebalance group telegraf_metrics_consumers in state PreparingRebalance with old generation 239 (__consumer_offsets-6) (reason: Adding new member Telegraf-b1168722-a4f9-42cc-b0c3-10c78e2cf4d1 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:01,426] INFO [GroupCoordinator 0]: Stabilized group telegraf_metrics_consumers generation 240 (__consumer_offsets-6) (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:01,441] INFO [GroupCoordinator 0]: Assignment received from leader for group telegraf_metrics_consumers for generation 240 (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:11,459] INFO [GroupCoordinator 0]: Member Telegraf-b1168722-a4f9-42cc-b0c3-10c78e2cf4d1 in group telegraf_metrics_consumers has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:11,459] INFO [GroupCoordinator 0]: Preparing to rebalance group telegraf_metrics_consumers in state PreparingRebalance with old generation 240 (__consumer_offsets-6) (reason: removing member Telegraf-b1168722-a4f9-42cc-b0c3-10c78e2cf4d1 on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:11,459] INFO [GroupCoordinator 0]: Group telegraf_metrics_consumers with generation 241 is now empty (__consumer_offsets-6) (kafka.coordinator.group.GroupCoordinator)
[2021-04-12 08:27:31,687] INFO [GroupCoordinator 0]: Dynamic Member with unknown member id joins group telegraf_metrics_consumers in Empty state. Created a new member id Telegraf-96ac6751-4f2e-4a67-80ee-1175c30540e4 for this member and add to the group. (kafka.coordinator.group.GroupCoordinator)
I use the following telegraf configuration for the kafka_consumer inputs plugin:
[[inputs.kafka_consumer]]
## Kafka brokers.
brokers = ["KAFKA_IP:9093"]
## Topics to consume.
topics = ["telegraf"]
## SSL parameters.
insecure_skip_verify = true
## Data format
data_format = "influx"
- Kafka version: 2.13-2.6.1
- Telegraf version: 1.18.0
When I run a Telegraf kafka_consumer inside the network and access Kafka server via its internal IP it works as expected. In that case, I am trying to access Kafka server via its public IP. Any idea where the issue could be in that case?
