How to get messages published to Redis before subscribing to the channel

Question

I am writing an application to get messages published to a channel in Redis and process them. It is a long-lived app which never basically listens to the channel.

def msg_handler():
    r = redis.client.StrictRedis(host='localhost', port=6379, db=0)
    sub = r.pubsub()
    sub.subscribe(settings.REDIS_CHANNEL)
    while True:
        msg = sub.get_message()
        if msg:
            if msg['type'] == 'message':
                print(msg)
def main():

    for i in range(3):
        t = threading.Thread(target=msg_handler, name='worker-%s' % i)
        print('thread {}'.format(i))
        t.setDaemon(True)
        t.start()
    while True:
        print('Waiting')
        time.sleep(1)

When I run this program, I notice that it is not getting messages that were published to the channel before the the program started. It works fine to get messages sent to the channel after the app subscribing to the channel.

In production, it is very likely there are some messages in the channel before the program starts. Is there a way to get these old messages?

The snippet does not run as necessary stuff is missing. What is `settings.REDIS_CHANNEL`?. You start three listeners but never send a message. Could you mock the filled REDIS_CHANNEL? — chriopp, May 17 '18 at 21:39
It is quite normal the `producer` of messages don't live in the same application as the `consumer` which is exactly the case here. I did not include the code publishing messages because it is working perfectly fine. And `REDIS_CHANNEL` is obviously just a constant. Why does it matter what it is? — ddd, May 17 '18 at 21:47
Because I can not reproduce your problem. You may want to dig in the code of pubsub to find out why you get no messages. — chriopp, May 17 '18 at 22:12
@Christoph I'd be surprised if you published before subscribed and still getting the messages. As I read more on this, it seems like Redis does not `Guarantee delivery`, which means "If a subscriber is not listening while a publication occurs, the event will be lost for this subscriber." according to [link](https://stackoverflow.com/questions/23675394/redis-publish-subscribe-is-redis-guaranteed-to-deliver-the-message-even-under-m) — ddd, May 18 '18 at 00:22

score 5 · Accepted Answer · answered May 18 '18 at 06:07

5

Redis PUB/SUB does not store messages that were published. It sends them to who was listening at the moment. If you need to have access to old messages, you can:

Use Redis Streams. They are in beta now and coming for version 5.
Use another PUBSUB system, for example nats.io

answered May 18 '18 at 06:07

Imaskar

2,773
24
35

Would RabbitMQ be a better option in this case? – ddd May 18 '18 at 16:19
I don't use RabbitMQ myself, but according to this question: https://stackoverflow.com/questions/33546568/how-do-i-get-old-messages-from-rabbitmq RabbitMQ can be configured to get prevoius messages. Idk how far can that go. In nats-streaming you have a lot of control over that. – Imaskar May 18 '18 at 16:27

How to get messages published to Redis before subscribing to the channel

1 Answers1