2

I am writing a clustered application sitting on top of Kafka -- it uses Kafka exclusively for interprocess communications and coordination. I could use Zookeeper to manage my cluster -- but it would not be very difficult to use Kafka topics to manage the cluster. And the more I think about it, other than for historical reasons, it seems like Kafka could drop Zookeeper and just use a topic-based solution

For example, there could be a special topic or topics in Kafka where you publish all of the same data currently kept track of in Zookeeper. Brokers, Topics, Partitions, Leaders, etc -- seems like this is just as easily tracked via Kafka topics as via Zookeeper.

I know in Kafka 0.9.0 there's some movement away from Zookeeper, more towards this model, and remember my question is less about Kafka development or more me trying to figure out which direction to go in my application.

I'm not asking for an opinion -- what I want to know is are there any specific functions provided by Zookeeper that are going to be difficult with a Kafka/topic-based approach to coordination. But I can't think of anything.

Even heartbeat monitoring -- which was the reason I started looking at Zookeeper in the first place -- you could have a client connection topic, and clients could publish to it when they join the cluster, publish heartbeats at a given interval, and publish as they leave it.

David Griffin
  • 13,677
  • 5
  • 47
  • 65

2 Answers2

3

Let us start from a space eyed view: You have two distributed systems which store data. Zookeeper organizes it's data in nodes in some kind of directory like structure. Kafka stores messages within topics.

From a bird eye view kafka is build for high-throughput and scalability while one of zookeepers main design goal is consistency. Zookeeper is mean to be a a Distributed Coordination Service for Distributed Applications while Kafka can be thought as a distributed commit log.

So the answer to your question is surprisingly: 'It depends'. For coordinating a distributed system I would use zookeeper: Thats what it was build for. You could do this also with kafka but there are couple of things which needs to be done manualy which comes out of the box if you are using zookeeper.

Some examples:

  • Consistency: The ZK-Client can choose if he needs strong or a eventual consistency
  • Ephemeral nodes: Together with ZK-Watches a great thing to react on failing services
  • Sequential Consistency: It's not granted that you recieve the kafka-message in the order you wrote it to the broker (it's only granted that messages within a partion are ordered)
  • ACLs: Never used it but its at least something which is not offered out of the box by kafka
  • Sequence Nodes

A pretty nice overview about what you can do with zookeeper are the zookeeper-recipes: https://zookeeper.apache.org/doc/trunk/recipes.html

[EDIT]: Heartbeating an application using kafka is of course possible. But ephemeral nodes in zookeeper are in my eyes the easier option.

Community
  • 1
  • 1
TobiSH
  • 2,833
  • 3
  • 23
  • 33
  • I am already using Kafka to coordinate the system -- worker nodes receive work requests via Kafka and write the results back. It seems like ZK just adds another layer to fail. Can you explain what you mean by strong vs eventual consistency? Sequential consistency is easy -- you gave the answer, just make sure everything you want is in one partition. Likewise I can also mimic both sequence and ephemeral nodes with a topic-based solution. Really, the ZK approach is like a relational model solution, where I can do the same thing with an event sourcing solution using topics. – David Griffin Mar 08 '16 at 18:11
  • If you argue that you can do everything with a kafka only setup than I would agree. You should just make sure that not everything looks like a nail just because you have a hammer. The argument that you don't want another system to maintain is a good one. But right now you still need a ZK to run a kafka cluster (even if it seems less impotant in kafka 0.9.x) - so why not use it? As I said there are solutions available in ZK which came for free - why rebuild it in Kafka? Could you please give some more insights what you are doing (beside the heartbeat example)? – TobiSH Mar 08 '16 at 20:12
  • I'm building a distributed app server in Node that uses event sourcing as the central data model. It's design goal is to handle millions of events per second and be able to perform complex, multi-step transformations in real-time. Things like strong consistency I had to solve anyway. The event sourcing way is totally different than the transactional model. Instead of locking a specific row to modify the current state of something, you just throw the updates into an ordered log, the results are published in real time to all the consumers. That's an over simplification, but you get the idea. – David Griffin Mar 08 '16 at 20:39
  • That sounds like a perfect use case for kafka. If you build your entire model on event sourcing than zookeeper might be not the right choice. Maybe zookeeper can help you with some "meta-tasks" like leader elections or two phase commits. Right tools for the right tasks! – TobiSH Mar 08 '16 at 21:30
  • Long story short, I'm going to have to deal with zookeeper because my app needs to respond to Kafka brokers leaving and joining. It sucks to have to learn a different metaphor, but there's no other way. For cluster management of my app I'm going to start with Zookeeper. App nodes leaving and joining the cluster -- that seems bread and butter Zookeeper. Once cluster management is stable, I can think about removing the dependency on Zookeeper from the app. If I even want to at that point. – David Griffin Mar 09 '16 at 19:36
0

This is currently being worked on in scope of KIP-500.

Kamil
  • 155
  • 1
  • 8