0

I have an Apache Geode setup, connected with external Postgres datasource. I have a scenario where I define an expiration time for a key. Let's say after T time the key is going to expire. Is there a way so that the keys which are going to expire can make a call to an external datasource and update the value incase the value has been changed? I want a kind of automatic syncing for my keys which are there in Apache Geode. Is there any interface which i can implement and get the desired behavior?

John Blum
  • 7,381
  • 1
  • 20
  • 30
user956021
  • 163
  • 1
  • 1
  • 12

1 Answers1

1

I am not sure I fully understand your question. Are you saying that the values in the cache may possibly be more recent than what is currently stored in the database?

Whether you are using Look-Aside Caching, Inline Caching, or even Near Caching, Apache Geode combined with Spring would take care of ensuring the cache and database are kept in sync, to some extent depending on the caching pattern.

With Look-Aside Caching, if used properly, the database (i.e. primary System of Record (SOR), e.g. Postgres in your case) should always be the most current. (Look-Aside) Caching is secondary.

With Synchronous Inline Caching (using a CacheLoader/CacheWriter combination for Read/Write-Through) and in particular, with emphasis on CacheWriter, during updates (e.g. Region.put(key, value) cache operations), the DB is written to first, before the entry is stored (or overwritten) in the cache. If the DB write fails, then the cache entry is not written or updated. This is true each time a value for a key is updated. If the key has not be updated recently, then the database should reflect the most recent value. Once again, the database should always be the most current.

With Asynchronous Inline Caching (using AEQ + Listener, for Write-Behind), the updates for a cache entry are queued and asynchronously written to the DB. If an entry is updated, then Geode can guarantee that the value is eventually written to the underlying DB regardless of whether the key expires at some time later or not. You can persist and replay the queue in case of system failures, conflate events, and so on. In this case, the cache and DB are eventually consistent and it is assumed that you are aware of this, and this is acceptable for your application use case.

Of course, all of these caching patterns and scenarios I described above assume nothing else is modifying the SOR/database. If another external application or process is also modifying the database, separate from your Geode-based application, then it would be possible for Geode to become out-of-sync with the database and you would need to take steps to identify this situation. This is rather an issue for reads, not writes. Of course, you further need to make sure that stale cache entries does not subsequently overwrite the database on an update. This is easy enough to handle with optimistic locking. You could even trigger a cache entry remove on an DB update failure to have the cache refreshed on read.

Anyway, all of this is to say, if you applied 1 of the caching patterns above correctly, the value in the cache should already be reflected in the DB (or will be in the Async, Write-Behind Caching UC), even if the entry eventually expires.

Make sense?

John Blum
  • 7,381
  • 1
  • 20
  • 30
  • John thanks for the answer but our deployment topology is like some of my application uses apache geode which interacts to postgres but there are few applications which directly updates postgres so how can geode know that those data are updated. Is there any way so that i can configure geode so that it will sync its values from postgres whch are updated by other applications? – user956021 May 23 '21 at 05:32
  • If there are applications that do not use Apache Geode and instead directly update the Postgres database in your overall system architecture, then your options, outside of a proprietary solution, are limited. Apache Geode, nor Spring, offer any support for this capability OOTB. – John Blum Jun 02 '21 at 18:15
  • However, you could implement a fixed "schedule", polling mechanism, such that you query the database for any changes in values associated with keys that are also stored in the database. I would suggest maintaining auditing metadata (e.g. version number) with the entity so you know the value has been changed in the database by another non-Apache Geode application. – John Blum Jun 02 '21 at 18:26
  • It would be simple enough to implement the fixed schedule polling mechanism with *Spring's* **Scheduling** support (https://docs.spring.io/spring-framework/docs/current/reference/html/integration.html#scheduling-annotation-support). You can read about *Spring Data's* **Auditing** support here (https://docs.spring.io/spring-data/commons/docs/current/reference/html/#auditing.basics). – John Blum Jun 02 '21 at 18:28
  • Ideally, the cache is "notified" when the data in the database is changed by some external, non-Apache Geode based application. For example, if you could write a Java-based Postgres Trigger to update the cache when a record changes and that record is also currently in the cache. Of course, all of this logic is tricky to implement correctly and efficiently, which is also why there is no general solution here. – John Blum Jun 02 '21 at 18:51