Redis Streams Partitioning

Question

I am trying to understand how redis streams does partitioning , if a specific message can be sent to a specific partition (similar to how you can do with Kafka).

I have checked the redis-cli api , and there is nothing similar to partitioning , also nothing about this when using the StackExchangeRedis redis library.

The only method is : IDatabase.StreamAdd(key,streamKey,streamValue,messageId)

Am i missing anything ? Is the partitioning done only in a fixed way ?

P.S If partitioning can be done , can the partitioning key be composed ?

https://redis.io/topics/streams-intro hope it can help. It also describes differences between Kafka and Redis partitions — maradev21, Oct 07 '21 at 14:08
Check this https://redis.io/topics/streams-intro#differences-with-kafka-tm-partitions. — sazzad, Oct 07 '21 at 14:13
There's a lib (written in Python) that does something similar: https://github.com/mjwestcott/runnel — lowercase00, May 14 '22 at 01:18

score 5 · Accepted Answer · answered Oct 08 '21 at 01:14

You cannot achieve Kafka partition with a single Redis Stream. Redis Stream's consumer group looks like a load balancer, which distributes messages in the stream to different consumers without any partition. If one consumer is faster than others, it will consume more messages than others, regardless of the logical partition of messages.

If you want to do partition, you have to use multiple Redis Streams, i.e. multiple keys, and each key for a different partition. And your producers add messages in a logical partition to a dedicated Redis Stream. The different consumers xread messages from different key.

Redis Streams Partitioning

1 Answers1