0

I have a nodejs application running (haraka smtp server) with 1 master thread and 7 worker threads . I'm using redis to read some counters and after some operations I'm updating those counters.

I'm facing the issue of race condition where some of the threads are reading the counters before other threads actually updates it, hence there is a mismatch in counters. Ideally it should be like one thread do its read and write operation on that particular key and then others should proceed with there ops.

I have currently a single instance of redis running. Is it a way to avoid this race condition with multiple threads and single redis instance ?

I've read about redlock but redis website says its best to use redlock with atleast 3 redis instances.

Dishant Kumawat
  • 1
  • Did you consider using redis pub/sub ? This way you could trigger node reads from other nodes. – JeanJacquesGourdin Sep 26 '22 at 14:32
  • Hi @JeanJacquesGourdin I'm new to the redis , can you guide me through it. – Dishant Kumawat Sep 27 '22 at 01:35
  • I've gone throught the redis pub/sub but I don't get how to use that in my particular use case where each node is both a publisher and subscriber. In my case what is actually happening is I'm storing ip addresses and some counters to the redis so to limit the use of those ips to a particular count. Now when an email request comes it checks with current count and increase that when it uses the ip but in case of multiple threads read of counts at the same time can cause a mismatch of count . – Dishant Kumawat Sep 27 '22 at 03:53
  • I guess i misunderstood what you wanted. How is triggered the email request ? Does this event triggers a process that can only be done by multiple workers ? What is the purpose of the count ? – JeanJacquesGourdin Sep 27 '22 at 06:15
  • Hi @JeanJacquesGourdin I'm running a haraka mail server which will listen for any email request comes to it and send it to a particular domain like gmail or yahoo, In the server I'm binding my custom IPs and sending it to its destination , So now I want to bind a particular IP only a fixed number of times – Dishant Kumawat Sep 27 '22 at 11:08
  • Now redis is used to store the IPs and counter ie to count how many times an ip is used. Here is the flow how it all works : email request comes , check the counter if the current ip is available to bind if yes bind it and increment the count else do not bind Here I'm using pm2 to run the server which is running it on 8 workers so when two or more threads reads the counter at the same time that's where the problem arises. – Dishant Kumawat Sep 27 '22 at 11:08

1 Answers1

0

For your use case, I dont think there is way to totally exclude read conflicts, but you can reduce the time window where data is inconsistent by doing the read/write on counters before doing the actual process, you could use a redis stream to be sure that every request is processed, processed only once and reduce the time window for inconsistencies.

It could work this way :

  1. An email request comes
  2. The worker INCR the key, it returns the new value of the counter
  3. If the counter is too large, do your stuff
  4. Else the worker pushes the event and its counter on a redis stream
  5. The redis stream distributes (fan out) events to workers via a consumer group
  6. A worker (maybe the same maybe another) is handed the event and has to process it
  7. Once it is processed its tell XACK to the redis stream to ensure the event has been processed

Edit:

Reading again this way of doing it appeared to me that the essential part is to use the INCR as a way to get the new value. It ensures that every request gets treated with a different counter. The redis stream is nice and all but unnecessary if you want to do it easily.

JeanJacquesGourdin
  • 1,496
  • 5
  • 25