7

In our architecture we have a Redis server we use for caching and for publishing event.

My problem is the following

  • I have an message called "CustomerUpdate"
  • I have 1 application listening to this message
  • 3 instance (server) of this application are being executed for scalability
  • 1 instance of the database is running
  • One of the handler for this message will update the database
  • Some other handler will erase memory cache or do something local to the instance

Is there any pattern for making sure that the database is not updated by every instance of the application?

gnat
  • 6,213
  • 108
  • 53
  • 73
remi bourgarel
  • 9,231
  • 4
  • 40
  • 73
  • What is the data structure you use in redis to achieve this? Queue or pub sub? – Karthikeyan Gopall Jul 25 '16 at 17:50
  • pub sub, there is no queue (afaik) in redis – remi bourgarel Jul 26 '16 at 07:40
  • Why use redis for the task it is not designed for? What you describe is one of the features supported by messaging buses. For instance, rabbitmq would suit perfectly here: https://www.rabbitmq.com/tutorials/tutorial-two-java.html In fact, rabbit would also be faster if performance is of any relevance. – Oleg Sklyar Aug 01 '16 at 12:52
  • @OlegSklyar because we already push messages to our redis instance from our apps, and I don't really want to have X messaging technologies. And here I don't want to create a work queue : "Some other handler will erase memory cache or do something local to the instance", with a queue, the other handler owuldn't receive the CustomerUpdate event – remi bourgarel Aug 01 '16 at 12:56
  • redis is not a messaging technology, so it is a lame argument. the result of misusing the technology will be your application will more error prone and less stable – Oleg Sklyar Aug 01 '16 at 12:58
  • @OlegSklyar thanks for your answer, but my question was not "how to do this with redis" but "how to do this", so ... do you have to be rude or you think it will improve your technical answer ? And you don't know me, so don't presume you know my technical skills. And from the redis home page "Redis is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker." – remi bourgarel Aug 01 '16 at 13:01
  • 1
    I am puzzled where exactly I was rude by posting the above comments. But the fact is the number of actual answers, and none of my comments was an answer, is indicative of how commonly redis is used to perform the task you describe. Good luck sticking to poor design even if this is rude of mine. – Oleg Sklyar Aug 01 '16 at 13:05
  • 1
    Please, put an answer starting with "redis is not the good tool for you, you can use XXX and do YYY". I'm not trying to figure out if my design decision are good or not. I'm just trying to know a "pattern" or a "tool" for making something. – remi bourgarel Aug 01 '16 at 13:07
  • @remibourgarel Does Redis's [Reliable Queue pattern](http://redis.io/commands/rpoplpush#pattern-reliable-queue) work for you? – heenenee Aug 01 '16 at 16:24
  • 1
    Here is your answer : http://stackoverflow.com/questions/7196306/competing-consumer-on-redis-pub-sub-supported – DhruvPathak Aug 02 '16 at 07:22
  • @heenenee, DhruvPathak , combining these 2 solutions with keyspace notifications on the list would definitely work, I'll try to make a small c# executable that does this synchronization work – remi bourgarel Aug 02 '16 at 07:28
  • @remibourgarel Great, please post some of your code as an answer whenever it's ready so as to help future visitors. – heenenee Aug 02 '16 at 12:19
  • http://programmers.stackexchange.com/questions/326296/what-pattern-should-i-set-for-handling-event-as-working-item/326300?noredirect=1#comment694110_326300 , I'll post it there, it seems more appropriate – remi bourgarel Aug 02 '16 at 12:40
  • I would advise an ORM. It will catch your multiple active result sets(MARS). The pattern is called "asynchronous Operations". https://msdn.microsoft.com/en-us/library/zw97wx20(v=vs.110).aspx –  Aug 05 '16 at 18:20

4 Answers4

6

You can use a redis key/value as blocker. When instances receive message from subcription excute LUA script in redis to check if process already exist for it.

Server receive message from subscription Use redis script transaction to check if there already exist a lock for this message (something Like get receiveMessageId:XXX). If value already exit with false then do nothing on server. If the value doesn't exist set it and return true. Then your server can process the message.

As Redis is single threaded all other server will get a false if message is taken by an other server.

To remove this key you can set a TTL big enought to avoid taken message from other servers.

khanou
  • 1,344
  • 11
  • 15
  • What if the Message id is not unique?? You will end up processing 1 event for n similar messages. – Karthikeyan Gopall Aug 04 '16 at 13:45
  • @khanou indeed it's a good idea, as long as you don't use a cluster, but it's not the case here. The obligation to add a unique message id is a bigger problem though : events in redis are just a message so you can't add meta data such as these (but you can publish a serialized "EventData" that would contain all the needed informations though). – remi bourgarel Aug 04 '16 at 15:11
3

A much simpler idea

1) Instead of publishing your events to a channel "CustomerUpdate" put them in a queue with a unique notifier notifying the type of action it is

LPUSH CustomerUpdate type1$somework

Here type1 can be sending email, entry in db, etc. and somework is the work that you need to handle.

2) In your application logic, use a blocking rpop.

BRPOP CustomerUpdate

in your app logic split the type of work and the work. if it returns type1 do that action, if it return type2 do that and so on. Then carry that work.

Sample snippet:

String message = jedis.brpop("CustomerUpdate",1000);
if(message.startsWith("type1$"))
sendMail(message.split("$")[1]);
else if(message.startsWith("type2$"))
sendAck(message.split("$")[1]);

Pros:

  • No need of keyspace notifications
  • Even if the app server is down you can do the works as in queue when it gets restarted, Whereas in pub/sub you can't get the messages already published.
  • Even if redis is down for few secs, you can get the data when persistence is enabled
  • Simple datastructre ( just a single queue )
Karthikeyan Gopall
  • 5,469
  • 2
  • 19
  • 35
  • With this solution, I have to update my publisher everytime I have a new type haven't I ? – remi bourgarel Aug 04 '16 at 09:53
  • Yes you have to. Anyhow you are going to add an implementation for it at the consumer end rit? During that time you can update the publisher as well. – Karthikeyan Gopall Aug 04 '16 at 10:02
  • The problem is not the coding effort. The problem is the coupling. We use events for reducing coupling (the module managing the customers data doesn't know about the email module for instance), if it now knows about all the other modules, then using the events is less interesting . – remi bourgarel Aug 04 '16 at 15:08
  • Just curious to know how will you handle this with a single event or pub/sub channel? Subscribers will have the recieved message string alone. how will you identify which module it belongs to? There should be some validatior in the consumer side to detect that rit? Correct me if I am wrong. – Karthikeyan Gopall Aug 04 '16 at 15:44
  • You don't have to identify who raised a business event, that's the whole point of using event : publisher and subscriber don't know about each others, they just know about the message broker it enable you to have a very loose coupling between your modules. Someone who want to do something when a customer is updated will just have to know about the event and ignore all the modules that publish this event (or the other subscribers) – remi bourgarel Aug 04 '16 at 15:57
  • Sorry, but again the same question coming in how will it know about that event in particular and ignore the other without any validation? – Karthikeyan Gopall Aug 04 '16 at 16:01
  • Same thing can be done with this model as well rit? Every pop is an event name with that event name you can proceed as such – Karthikeyan Gopall Aug 05 '16 at 12:11
2

A simplistic way of doing it, rather than sending in the message the event data, send the name of the list that contains such data, then the first receiver of the message will execute a LPOP on such list and only it will receive the event data.

In a nutshell:

  • your clients subscribe like SUBSCRIBE CustomerUpdate.
  • the publisher executes RPUSH CustomerUpdateList <data>; PUBLISH CustomerUpdate CustomerUpdateList.
  • all the clients will get a MESSAGE CustomerUpdate CustomerUpdateList, but only the first one executing LPOP CustomerUpdateListwill get the message <data>.

However, from the moment you execute the LPOP in the server, the message will be processed or will be lost forever. If for example the connection drops right after the LPOP, the message will be lost.

Implementing reliable messaging in Redis is hard, so you may be better taking a look at projects like : https://github.com/resque/resque or https://github.com/seomoz/qless

Or if you want to do it yourself, take a look to this presentation where the authors give a good explanation about the approach they followed : https://www.percona.com/news-and-events/percona-university-smart-data-raleigh/using-redis-reliable-work-queue

PS: Although my recommendation would be to get something like RabbitMQ for this kind of things.

vtortola
  • 34,709
  • 29
  • 161
  • 263
0

I'd set up one or more lists as queues for the actionable tasks for CustomerUpdate. Instead (or as well as) publishing the CustomerUpdate, you would LPUSH onto the list. The value of each of the list elements would encode the parameters of the update.

Then simply use BRPOP in a loop in each of the handlers that need to contend for the jobs. This is a blocking pop with a timeout, designed for use cases like this. http://redis.io/commands/brpop

Pros: no keyspace notifications, many job handlers can pop without races, can separate necessary tasks into separate lists, and BRPOP multiple lists at once.

Cons: whatever publishes the CustomerUpdate would need to be changed, and possibly do multiple LPUSHs, possibly needing MULTI/EXEC or similar. If you can't change this aspect, you'd need another process (different client) to subscribe to CustomerUpdates, and push the jobs.

nnog
  • 1,607
  • 16
  • 23