Understand Kafka write speed

Question

Given that
1. Five machine Kafka cluster
2. 1 topic with 1 partition
3. persisted storage for msgs
4. Each msg of 1 KB
5. 10 producers

Now, Will the max write capability of this cluster BE EQUAL TO max write capability of one java thread onto a disk? If No, what will it be?

Kafka writes to memory before flushing to disk. Not sure why you wouldn't want 5 partitions if you're concerned about speed — OneCricketeer, Jun 06 '18 at 13:52
This is a academic question to understand and create mind model..the case I am analying required persistence — Bhuvan, Jun 07 '18 at 06:28
This is a bad question. Showing close-to-zero previous research efforts. A few strokes and a few minutes of reading will avoid asking artificially "constructed" question, even from academic residents ... >>> https://www.quora.com/Kafka-writes-every-message-to-broker-disk-Still-performance-wise-it-is-better-than-some-of-the-in-memory-message-storing-message-queues-Why-is-that + a few google refinements will get one more inputs on **Why ?** - if indeed interested in and striving to **create a mind model**. — user3666197, Jun 12 '18 at 06:37

score 1 · Accepted Answer · answered Jun 12 '18 at 06:22

if you only have one partiton then kafka cannot scale your topic and use only one machine out of your 5 instance cluster.

cannot tell how well you code your java but results will be similar to kafka I assume with small differences as kafka is optimised for disk writes.

score 1 · Answer 2 · edited Jun 12 '18 at 06:34

1

You can't really say it will be "EQUAL TO" because it is not accurate, but yes, the write capacity will be pretty much the same assuming the disk I/O will be your bottle-neck. Take into account that Kafka is optimized for this task and probably will better perform than the naive disk writes of a standard Java application.

edited Jun 12 '18 at 06:34

user3666197

1
6
50
92

answered Jun 12 '18 at 06:33

Roee Gavirel

18,955
12
67
94

Understand Kafka write speed

2 Answers2