Stack Overflow
Stack Overflow

Questions tagged [redis-streams]

Redis Streams is a data structure released as part of Redis 5.0. Conceptually inspired by Apache™ Kafka, they are a log-like structure designed for storing append-only semi-structured data. Use this tag for questions related to the Redis Stream data type.

Redis Streams is a data structure released as part of Redis 5.0. Conceptually inspired by Apache™ Kafka, they are a log-like structure designed for storing append-only semi-structured data.

88 questions
22
votes
1 answer

Redis Streams vs Kafka Streams/NATS

Redis team introduce new Streams data type for Redis 5.0. Since Streams looks like Kafka topics from first view it seems difficult to find real world examples for using it. In streams intro we have comparison with Kafka streams: Runtime consumer…
Nick Bondarenko
  • 6,211
  • 4
  • 35
  • 56
12
votes
1 answer

Node Redis XREAD blocking subscription

I'm converting a redis pub/sub system to redis streams, so that I can add some fault tolerance to my server sent events. Subscribing the traditional way is trivial: import { createClient } from 'redis'; const redisOptions = { url:…
Dudo
  • 4,002
  • 8
  • 32
  • 57
10
votes
1 answer

How to prevent Redis stream memory increases infinitely?

I just realized that the XACK do not auto delete message when only one consumer group exist. I thought that when all consumer groups ack the same message, the message will be deleted by Redis-server, but seemed that this is not the case. So, the…
UNSTABLE
  • 393
  • 1
  • 4
  • 13
7
votes
4 answers

How to continuosly listen on redis stream using lettuce java library

I am trying to listen on a redis stream and process the message as and when they arrive. I am using async command and I expect the message to be pushed instead of being pulled. So I don't think a while loop is required. But the following code seems…
falcon
  • 1,332
  • 20
  • 39
6
votes
1 answer

Count or list all ACKnowledged messages in a redis stream

I understand I can get the total length of a stream with xlen or xinfo stream mystream full. I also understand that I can use xpending to get the length of the pending queue, items that have not been ack'd. Is there a command to give me the count or…
Mittenchops
  • 18,633
  • 33
  • 128
  • 246
5
votes
1 answer

Is there a way to subscribe to Redis Streams similar to Redis pub/sub?

I am building a data pipeline that reads the MySQL binlogs in realtime using Maxwell Daemon and Maxwell Daemon pushes the data to Redis streams. I've built a client that consumes the data from Redis streams processes it. Currently, I'm polling the…
Utkarsh
  • 83
  • 2
  • 10
5
votes
3 answers

Convert Redis Streams output to Pandas Dataframe

What would be the fastest way to convert a Redis Stream output (aioredis client/ hiredis parser) to a Pandas Dataframe where Redis Stream ID‘s timestamp and sequence number as well as values are proper type converted Pandas index columns? Example…
trbck
  • 5,187
  • 6
  • 26
  • 29
4
votes
0 answers

Consuming messages from Redis Stream using Consumer Groups in NodeJs

Redis Consumer Groups is relatively a new feature in Redis and so I couldn't find a matured enough npm package that provide out of the box mechanism to read messages using consumer group. In Spring Data Redis, onMessage(V message) of StreamListener…
Uzair
  • 469
  • 3
  • 13
4
votes
1 answer

Redis streams in cluster mode

Do Redis streams benefit from Cluster mode? Imagine you had 10 streams, would they be distributed across the cluster or all on the same node? I'm planning on using Redis streams for really high throughput (2m+ messages/s) so I'm worried about the…
Gabriel Mendez
  • 61
  • 3
4
votes
2 answers

Redis streams - remove / expire events based on time

I am playing with redis stream and it is good so far. I am trying to understand if there is anyway for me to expire the old events based on time or some other way. I know that we can remove by event id. But I do not want to remember / store the…
RamPrakash
  • 2,218
  • 3
  • 25
  • 54
3
votes
1 answer

Redis Streams: How to manage perpetual subscription and BLOCK behaviour?

I am using Redis in an Express application. My app is both a publisher and consumer of streams, using a single redis connection (redis.createClient). I have a question on the best way to manage a perpetual subscription (with xreadgroup). Currently I…
jramm
  • 6,415
  • 4
  • 34
  • 73
3
votes
1 answer

Deleting a range of entries from a Redis Stream

I want to delete entries of a Redis Stream older than a particular entry ID. But the XDEL command takes each ID explicitly as input. Is there any way to specify a range of IDs which will help when there are large number of entries in the stream?…
aniztar
  • 2,443
  • 4
  • 18
  • 24
2
votes
0 answers

XACK is not deleting the message, even if it is processed successfully?

I am trying to implement redis stream where we have a producer. package producer import ( "RedisStream/models" "encoding/json" "fmt" "github.com/garyburd/redigo/redis" ) type Producer struct { streamName string } func…
Ashutosh Pandey
  • 120
  • 8
2
votes
0 answers

How does spring-data-redis get an objectRecord with a Generic type

I want to implement MQ using stream in Redis, and when the entity type passed is generic, my code does not deserialize the entity consumption message correctly. producer critical code: @Bean RedisTemplate…
Mornie alantie
  • 33
  • 2
2
votes
1 answer

Redis Streams Partitioning

I am trying to understand how redis streams does partitioning , if a specific message can be sent to a specific partition (similar to how you can do with Kafka). I have checked the redis-cli api , and there is nothing similar to partitioning , also…
Bercovici Adrian
  • 8,794
  • 17
  • 73
  • 152
1
2 3 4 5 6