Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

202 questions
55
votes
7 answers

How to programmatically create a topic in Apache Kafka using Python

So far I haven't seen a python client that implements the creation of a topic explicitly without using the configuration option to create automatically the topics.
jpgerek
  • 796
  • 1
  • 8
  • 13
37
votes
5 answers

How to delete multiple topics in Apache Kafka

Assuming that I have a number of topics with the same prefix, e.g: giorgos-topic1 giorgos-topic2 giorgos-topic3 ... The command used for deleting a single topic (say giorgos-topic1) is the following: ./bin/kafka-topics.sh --zookeeper localhost:2181…
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
27
votes
1 answer

Why is kafka not creating a topic? bootstrap-server is not a recognized option

I am new to Kafka and trying to create a new topic on my local machine. I am following this link. Here are the steps which i followed: Start zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties Start…
KayV
  • 12,987
  • 11
  • 98
  • 148
15
votes
1 answer

Removing one message from a topic in Kafka

I'm new at using Kafka and I have one question. Can I delete only ONE message from a topic if I know the topic, the offset and the partition? And if not is there any alternative?
Ander99
  • 163
  • 1
  • 1
  • 4
10
votes
2 answers

Can single consumer read from multiple partitions of a kafka topic?

There is a kafka topic with 16 partitions With a given consumer group name, we are currently launching single consumer to read from the topic. Does single consumer read from partition 0(only) of that topic? If partition 0 has gone empty with…
overexchange
  • 15,768
  • 30
  • 152
  • 347
9
votes
3 answers

Kafka log.segment.bytes vs log.retention.hours

I was following the book "Kafka: The Definitive Guide" First Edition to understand when log segments are deleted by the broker. As per the text I understood, a segment will not become eligible for deletion until it is closed. A segment can be closed…
Mayank Madhav
  • 429
  • 1
  • 7
  • 19
9
votes
4 answers

What is a partition leader in Apache Kafka?

Are kafka leaders partitions themselves or are they brokers? My initial understanding was that they were partitions which acted as read/write agents which then deffered their value to ISRs. However recently I have been hearing them mentioned as…
Matt
  • 113
  • 1
  • 1
  • 5
9
votes
1 answer

Kafka Streams Testing : java.util.NoSuchElementException: Uninitialized topic: "output_topic_name"

I've written a test class for kafka stream application as per https://kafka.apache.org/24/documentation/streams/developer-guide/testing.html , the code for which is import com.EventSerde; import org.apache.kafka.common.serialization.Serde; import…
Arbaz Sheikh
  • 217
  • 2
  • 10
9
votes
1 answer

AWS MSK - Timeout when creating Kafka topic with ACL turned-on

I'm using AWS MSK and I want to enable ACLs but I'm unable to create a topic when ACLs are turned-on. I'm using the command-line tools for all the operations. Here's a summary of what I'm doing: Create a fresh cluster Create a topic - this works…
Araf
  • 353
  • 5
  • 11
9
votes
3 answers

how many consumer groups can a kafka topic handle?

suppose I have a kafka topic with say about 10 partitions, I understand that every consumer group should have 10 consumers reading from the topic at any given time to achieve maximum paralellism. However, I wanted to know if there is any direct rule…
mythic
  • 535
  • 7
  • 21
9
votes
1 answer

Kafka topic has partitions with leader=-1 (Kafka Leader Election), while node is up and running

I have a 3 member kafka-cluster setup, the __consumer_offsets topic has 50 partitions. The following is the result of describe command on: root@kafka-cluster-0:~# kafka-topics.sh --zookeeper localhost:2181 --describe Topic:__consumer_offsets …
SpiXel
  • 4,338
  • 1
  • 29
  • 45
7
votes
0 answers

When does Kafka topic-level configuration changes take effect?

There are documents such as the ones below stating that Kafka topic-level configuration changes can be taken place while Kafka cluster is still…
Alex Fung
  • 1,996
  • 13
  • 21
6
votes
1 answer

Kafka-topics.sh authentication

I am learning Apache Kafka and I do not understand how to make kafka-topics.sh work with configured SASL_PLAINTEXT authentication on the server. This is a server.properties…
Alex.P
  • 63
  • 1
  • 3
6
votes
1 answer

How Kafka guarantee the messages order while we increase the partitions in runtime?

I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic…
injoy
  • 3,993
  • 10
  • 40
  • 69
6
votes
4 answers

Kafka consumer not consuming from beginning

I have Kafka setup on my local machine and have started the zookeeper and a single broker server. Now i have a single topic with following description: ~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-topics.sh --zookeeper 127.0.0.1:2181…
KayV
  • 12,987
  • 11
  • 98
  • 148
1
2 3
…
13 14