Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
358
votes
5 answers

Understanding Kafka Topics and Partitions

I am starting to learn Kafka. During my readings, some questions came to my mind: When a producer is producing a message, it will specify the topic it wants to send the message to. Is that right? Does it care about partitions? When a subscriber is…
Pinidbest
  • 3,866
  • 4
  • 15
  • 17
182
votes
3 answers

Is key required as part of sending messages to Kafka?

KeyedMessage keyedMessage = new KeyedMessage(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed…
gaurav
  • 2,886
  • 6
  • 24
  • 26
72
votes
10 answers

How to check whether Kafka Server is running?

I want to ensure whether kafka server is running or not before starting production and consumption jobs. It is in windows environment and here's my kafka server's code in eclipse... Properties properties = new…
User
  • 4,023
  • 4
  • 37
  • 63
57
votes
2 answers

Topics, partitions and keys

I am looking for some clarification on the subject. In Kafka documentations I found the following: Kafka only provides a total order over messages within a partition, not between different partitions in a topic. Per-partition ordering combined with…
Igor K.
  • 915
  • 2
  • 12
  • 22
52
votes
2 answers

How to send key, value messages with the kafka console producer

I got a use case where I need to send key value messages with Kafka Console Producer. So how to achieve this through the Kafka Console Producer command?
Keshav Lodhi
  • 2,641
  • 2
  • 17
  • 23
50
votes
4 answers

How to write a file to Kafka Producer

I am trying to load a simple text file instead of standard input in Kafka. After downloading Kafka, I performed the following steps: Started zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties Started…
Katie
  • 763
  • 1
  • 9
  • 21
40
votes
3 answers

Kafka - difference between Log end offset(LEO) vs High Watermark(HW)

What is the difference between LEO and HW in Replica ( Leader Replica)? Will they contain the same number? I can understand HW is the last committed message offset. When LEO will be updated and how?
Shankar
  • 8,529
  • 26
  • 90
  • 159
40
votes
5 answers

How to create topics in apache kafka?

What is the bestway to create topics in kafka? How many replicas/partitions to be defined when we create topics? In the new producer API, when i try to publish a message to a non existing topic , it first time fails and then successfully…
Ratha
  • 9,434
  • 17
  • 85
  • 163
35
votes
1 answer

heartbeat failed for group because it's rebalancing

What's the exact reason to have heartbeat failure for group because it's rebalancing ? What's the reason for rebalance where all the consumers in group are up ? Thank you.
AloneArtifact
  • 475
  • 1
  • 5
  • 8
34
votes
5 answers

KafkaProducer: Difference between `callback` and returned `Future`?

The KafkaProducer send method both returns a Future and accepts a Callback. Is there any fundamental difference between using one mechanism over the other to execute an action upon completion of the sending?
Thilo
  • 257,207
  • 101
  • 511
  • 656
33
votes
6 answers

Kafka Getting error No resolvable bootstrap urls given in bootstrap servers

I'm pretty sure the bootstrap.servers is correct. Is there anything confliction in Maven or what wrong with Kafka?? Before that It was working successfully. I added some Maven or Spark then something went wrong.. Can anyone knows how to resolve…
Vimal Dhaduk
  • 994
  • 2
  • 18
  • 43
32
votes
17 answers

org.apache.kafka.common.errors.TimeoutException: Topic not present in metadata after 60000 ms

I'm getting the error: org.apache.kafka.common.errors.TimeoutException: Topic testtopic2 not present in metadata after 60000 ms. When trying to produce to the topic in my local kafka instance on windows using Java. Note that the topic testtopic2…
Gherbi Hicham
  • 2,416
  • 4
  • 26
  • 41
32
votes
5 answers

how to view kafka headers

We are sending message with headers to Kafka using org.apache.kafka.clients.producer.ProducerRecord public ProducerRecord(String topic, Integer partition, K key, V value, Iterable
headers) { this(topic, partition, (Long)null, key,…
Sammy Pawar
  • 1,201
  • 3
  • 19
  • 38
32
votes
4 answers

Kafka producer TimeoutException: Expiring 1 record(s)

I am using Kafka with Spring-boot: Kafka Producer class: @Service public class MyKafkaProducer { @Autowired private KafkaTemplate kafkaTemplate; private static Logger LOGGER =…
Prakash P
  • 3,582
  • 4
  • 38
  • 66
31
votes
4 answers

Does Kafka support request response messaging

I am investigating Kafka 9 as a hobby project and completed a few "Hello World" type examples. I have got to thinking about Real World Kafka applications based on request response messaging in general and more specifically how to link a Kafka…
Hector
  • 4,016
  • 21
  • 112
  • 211
1
2 3
…
99 100