0

env

  • apache kafka 2.7
  • apache flume 1.9.0

trouble

I'm gonna setup kafka cluster with version 2.7.
Using DNS attribute(use_all_dns_ips), I want to perform traffic switching by dividing the dns of both clusters in case of DR.
But now, i using Apache Flume tailing source. I cannot apply DNS attribute values in flume because flume channel doesn't support DNS properties.

I also looked for the kafka source connector, but in the official version, there seems to be only a spooldir version and no tailing source version.
Is there any tailing source application that you can recommend to add a dns attribute?

hoon
  • 43
  • 1
  • 9
  • I don't think that property does what you think because offsets are still tracked _per cluster_, so there's no DR scenario where a consumer will recover correctly – OneCricketeer Jul 26 '21 at 13:53
  • 1
    Sure, but I was specifically talking about Flume – OneCricketeer Jul 26 '21 at 14:23
  • I'm gonna use dns per cluster. And MirrorMaker2 consumer offset sync. https://stackoverflow.com/questions/65925842/kafka-mirrormaker2-automated-consumer-offset-sync Is it an impossible theory? I referrenced this article too. https://dalelane.co.uk/blog/?p=4074 – hoon Jul 26 '21 at 14:25
  • Oh, i see. thanks always. – hoon Jul 26 '21 at 14:26
  • Flume ought to act as any regular producer/consumer client, as its Java code at the end of the day using `kafka-clients` dependency. That being said, you should be able to set any client properties you want, so what do you mean "doesn't support" it? – OneCricketeer Jul 26 '21 at 14:30
  • Yes, i checked this docs. https://issues.apache.org/jira/browse/FLUME-3281 Flume supports kafka client 2.0.1 version compatibility. But dns property works over 2.1.0 version. So I updated kafka-client to ver 2.1.0, but kafka-channel dns property doesn't work. – hoon Jul 27 '21 at 02:42
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/235320/discussion-between-hoon-and-onecricketeer). – hoon Jul 27 '21 at 04:14
  • I haven't really used Flume since like version 1.6... Filebeat, Logstash, Rsyslog, Fluentd, etc have all been used instead with Kafka where I've worked on rotating log files – OneCricketeer Jul 27 '21 at 14:34

0 Answers0