16

Say, I'm Fechting thousands or record using some long runing task from DB and caching it using Redis. Next day somebody have changed few records in DB.

Next time how redis would know that it has to return cached data or again have to revisit that all thousands of records in DB?

How this synchronisation achived?

Swapnil Kotwal
  • 5,418
  • 6
  • 48
  • 92

4 Answers4

20

Redis has no idea whether the data in DB has been updated.

Normally, we use Redis to cache data as follows:

  1. Client checks if the data, e.g. key-value pair, exists in Redis.
  2. If the key exists, client gets the corresponding value from Redis.
  3. Otherwise, it gets data from DB, and sets it to Redis. Also client sets an expiration, say 5 minutes, for the key-value pair in Redis.
  4. Then any subsequent requests for the same key will be served by Redis. Although the data in Redis might be out-of-date.
  5. However, after 5 minutes, this key will be removed from Redis automatically.
  6. Go to step 1.

So in order to keep your data in Redis update-to-date, you can set a short expiration time. However, your DB has to serve lots of requests.

If you want to largely decrease requests to DB, you can set a large expiration time. So that, most of time, Redis can serve the requests with possible staled data.

You should consider carefully about the trade-off between performance and staled data.

for_stack
  • 21,012
  • 4
  • 35
  • 48
  • Also check [this question](https://stackoverflow.com/questions/45384889/how-to-keep-caching-up-to-date/45385486). – for_stack Nov 24 '18 at 14:59
  • Also check [this](https://stackoverflow.com/questions/76518041/cache-aside-strategy-delete-or-update-the-cache-after-the-write/76519492#76519492) for how to make cache be consistent with database as much as possible. Although the solutions have their own problems. – for_stack Jun 21 '23 at 02:11
5

Since the source of truth resides on your Database and you push data from this DB to Redis, you always have to update from DB to Redis, at least you create another process to sync data.

My suggestion is just to run a first full update from DB to Redis and then use a synch process which every time you notice update/creation/deletion operation in your database you pull it to Redis.

I don't know which Redis structure are you using to store database records in Redis but I guess it could be a Hash, probably indexed by your table index so the sync operation will be immediate: if a record is created in your database you set a HSET, if deletion HDEL and so on.

You even could omit the first full sync from DB to Redis, and just clean Redis and start the sync process.

If you cannot do the above for some reason you can create a syncher daemon which constantly read data from the database and compare them with the data store in Redis if they are different in somehow you update or if they don't exist in some of both sides you can delete or create the entry in Redis.

Averias
  • 931
  • 1
  • 11
  • 20
  • see, in next time very few records are changed. Is it possible that I could get cached data from redis and only changed data from DB? – Swapnil Kotwal Mar 15 '18 at 13:29
  • Umm, I don't know if I understood you well, in Redis all data are cached, you only have to query the proper structure with the correct key (or member), regarding the database, depends on which one you have, for instance if you use MongoDB, they have that nice ``oplog`` which is a capped collection where all changes in the database are registered, so you can read the oplog in another process and regarding the operation type (delete, create or update) update accordingly Redis. If you are using another RDBS or No-SQL database we have to see if there is something similar to oplog – Averias Mar 15 '18 at 13:39
  • Or a good approach to avoid having another process reading from oplog or wherever (depending on the kind of DB you have) is every time that you update the database from your application you save in the database and also make the same operation in Redis. You can accomplish that by having DB listeners for every operation (update, delete and create registers) which will be in charge to update Redis – Averias Mar 15 '18 at 13:43
  • 1
    Here the link to ``oplog`` in case you use MongoDB https://docs.mongodb.com/manual/core/replica-set-oplog/ – Averias Mar 15 '18 at 13:50
  • 1
    aaah ok, I understand now what you asked above, you want to get data from Redis and for those register which changed it, take the data from the database, right? To be honest, then it doesn't make much sense to have Redis, the good thing to have Redis is you can update every time database change and you will always have updated data in Redis. Besides, you will need to be aware when a register change and which one to decide to take the data from Redis or DB. Anyway, let's say you cannot/don't want to take some of my proposals and you want to do in the way you asked in the comment... – Averias Mar 15 '18 at 14:48
  • What you could do is to have another structure in Redis, I think a SET would be fine, where you can store the register ID which was updated in the database and every time you have to take the info for that register look at the SET and if the id is there will mean the register was updated, so you go to the database to get the data. BUT this doesn't make sense, since, first, you still need to update that SET every time a register change in the DB and secondly, if you are gonna update the SET why don't update directly the the register data in the structure you already have in Redis – Averias Mar 15 '18 at 14:53
  • Thanks @Averias for all your help. It clarifies many things to me :) – Swapnil Kotwal Mar 15 '18 at 15:04
1

My solution is:

When you are updating, deleting or adding new data in database, you should delete all data in redis. In your get route, you should check if data exists. If not, you should store all data to redis from db.

Adham Muzaffarov
  • 56
  • 1
  • 6
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Oct 25 '21 at 05:46
0

you may use @CacheEvict on any update/delete applied on DB. that would clear up responding values from cache, so next query would get from DB

jeff
  • 1