Stack Overflow
Stack Overflow

Questions tagged [apache-zookeeper]

Apache ZooKeeper is a distributed service that provides configuration information storage, naming, synchronization, coordination, and group services.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. It was inspired by Chubby, Google's distributed lock system.

All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which makes them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

ZooKeeper is used by a variety of other software projects to maintain configuration information including Apache Storm, Apache Flink, Solr Cloud, Apache Kafka and Apache Hadoop.

Links

Related Tags

3686 questions
413
votes
8 answers

Explaining Apache ZooKeeper

I am trying to understand ZooKeeper, how it works and what it does. Is there any application which is comparable to ZooKeeper? If you know, then how would you describe ZooKeeper to a layman? I have tried apache wiki, zookeeper sourceforge...but I…
topgun_ivard
  • 8,376
  • 10
  • 38
  • 45
183
votes
4 answers

Data Modeling with Kafka? Topics and Partitions

One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?". I've read and watched some introductory materials. In particular, take, for example, Kafka: a…
David J.
  • 31,569
  • 22
  • 122
  • 174
171
votes
13 answers

Is Zookeeper a must for Kafka?

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using…
Paaji
  • 2,139
  • 4
  • 14
  • 11
124
votes
14 answers

Distributed sequence number generation?

I've generally implemented sequence number generation using database sequences in the past. e.g. Using Postgres SERIAL type http://www.neilconway.org/docs/sequences/ I'm curious though as how to generate sequence numbers for large distributed…
Jonathan Holloway
  • 62,090
  • 32
  • 125
  • 150
122
votes
13 answers

Real World Use of Zookeeper

I've been looking at Zookeeper recently and wondered whether anybody was using it currently and what they were specifically using it for storing. The most common use case is for configuration information, but what kind of data and how much data are…
Jonathan Holloway
  • 62,090
  • 32
  • 125
  • 150
111
votes
16 answers

Is there a way to delete all the data from a topic or delete the topic before every run?

Is there a way to delete all the data from a topic or delete the topic before every run? Can I modify the KafkaConfig.scala file to change the logRetentionHours property? Is there a way the messages gets deleted as soon as the consumer reads it? I…
TommyT
  • 1,707
  • 3
  • 17
  • 26
105
votes
18 answers

List all Kafka 0.10 topics using - -zookeeper flag without access to Zookeeper

I'm using kafka 0.10 without zookeeper. I want to get kafka topics list. This command is not working since we're not using zookeeper: bin/kafka-topics.sh --list --zookeeper localhost:2181. How can I get the same output without zookeeper?
SSR
  • 1,249
  • 2
  • 10
  • 12
95
votes
12 answers

How to check if ZooKeeper is running or up from command prompt?

I exploring a few options to setup kafka and I knew that the Zookeeper has to be up and running to initiate a kafka. I would like to know how can I find the below. 1) hostname and port for my zookeeper instance---I checked the zoo.cfg and I could…
Suren Baskaran
  • 1,228
  • 1
  • 10
  • 17
84
votes
4 answers

zookeeper is not a recognized option when executing kafka-console-consumer.sh

I'm learning how to use Kafka on this website link(except I'm using port 2182 in zookeeper), but it shows: zookeeper is not a recognized option after executing: sudo ./bin/kafka-console-consumer.sh --topic test --zookeeper localhost:2182 How to…
xingbin
  • 27,410
  • 9
  • 53
  • 103
77
votes
11 answers

ZooKeeper alternatives? (cluster coordination service)

ZooKeeper is a highly available coordination service for data centers. It originated in the Hadoop project. One can implement locking, fail over, leader election, group membership and other coordination issues on top of it. Are there any…
Thomas Koch
  • 2,833
  • 2
  • 28
  • 36
70
votes
21 answers

classpath is empty. please build the project first

I was trying to run Kafka on Windows machine and when I try to start the zookeeper I am facing this weird error: classpath is empty. please build the project first e.g. by running 'gradlew jarall'
Vishesh
  • 3,599
  • 4
  • 22
  • 36
67
votes
12 answers

Running into LeaderNotAvailableException when using Kafka 0.8.1 with Zookeeper 3.4.6

I installed the stable version of kafka (0.8.1 with 2.9.2 Scala) as per their website and am running it with a 3 node zookeeper ensemble (3.4.6). I tried to create a test topic but keep seeing that there is no leader assigned to the partition of the…
user3561789
  • 671
  • 1
  • 5
  • 4
66
votes
6 answers

supervisord stopping child processes

One of the problems, I face with supervisord is that when I have a command which in turn spawns another process, supervisord is not able to kill it. For example I have a java process which when runs normally is like $ zkServer.sh start-foreground …
FUD
  • 5,114
  • 7
  • 39
  • 61
63
votes
6 answers

Kafka + Zookeeper: Connection to node -1 could not be established. Broker may not be available

I am running in my locahost both Zookeeper and Kafka (1 instance each). I create succesfully a topic from kafka: ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partitions 1 --topic Hello-Nicola Created topic…
Nicola Ben
  • 10,615
  • 8
  • 41
  • 65
58
votes
3 answers

Zookeeper & Kafka error KeeperErrorCode=NodeExists

I have written a kafka consumer and producer that worked fine until today. This morning, when I started zookeeper and kafka, my consumer was not able to read messages, and I found this in the zookeeper logs: INFO Got user-level KeeperException when…
adellarocca
  • 649
  • 1
  • 5
  • 7
1
2 3
…
99 100