3

I am using the idempotent receiver pattern in my spring integration flow to detect duplicate messages.

https://docs.spring.io/spring-integration/docs/5.0.5.RELEASE/reference/html/system-management-chapter.html#metadata-store

In above link it says

The value of the idempotent entry may be some expiration date, after which that entry should be removed from Metadata Store by some scheduled reaper.

Is there any utility/scheduler available that is provided by spring to clean up the INT_METADATA_STORE table ?

nagendra
  • 593
  • 1
  • 9
  • 29

1 Answers1

2

Well, there is no such a built-in tool for the MetadaStore clean up, although I believe we need to add: that's not a first time when we get a request to be able to delete (or expire) some old entries in the store. Feel free to raise a JIRA on the matter. Looks like we need to add some extra column: UPDATE_DATE to be able to compare it with some time from the expireUnusedOlderThan(long age) callback. Something similar what we have with the ExpirableLockRegistry.

Since you talk about an INT_METADATA_STORE table, I believe you can use for time being a plain JdbcTemplate and some scheduled task from the Spring Framework: https://docs.spring.io/spring/docs/current/spring-framework-reference/integration.html#scheduling-annotation-support

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • Thanks will raise Jira, i am storing the jms_messageId header value in METADATA_KEY & I see timestamp in milliseconds is stored in METADATA_VALUE automatically. I changed the METADATA_VALUE column type in database to "bigint" from "varchar" and wrote a scheduler to delete the records older than one day. – nagendra Aug 14 '18 at 16:15
  • I've encountered the same problem. For some reasons I store status of message in the column **metadata_value**. So it might be **started** or **finished** for now. I need to cleanup "finished" entries which were finished more than 1 day ago. So I need 1 additional column with **UPDATE_DATE** as you mentioned in the answer. Is there any workaround for now? I expect huge amount of messages so value like **finished_timestamp** doesn't look like a appropriate option. – gstackoverflow Dec 25 '19 at 14:50
  • For me "started", "finished" and if I don't have entry at all - 3 different statuse which should be handled differently. – gstackoverflow Dec 25 '19 at 14:54
  • I don't think the `MetadaStore` is an appropriate pattern for your use-case. The "started", "finished" and so on don't sound like a metadata. I believe you need to have your own table structure and handle it yourself using the mentioned `JdbcTemplate`. Or Spring Integration JDBC Channel Adapters can help you as well. My point is let's don't abuse components for particular patterns in any other possible places! – Artem Bilan Dec 25 '19 at 14:57
  • @Artem Bilan we had to do it because sometimes messages from Google pub sub are not removed from subscription even after acknowldge(see https://stackoverflow.com/questions/59391264/message-is-received-from-google-pub-sub-subscription-again-and-again-after-ackno). So I believe it is correct usage of idempotency store. It might be an option moving entries from MetaDataStore table to the our own table with adding timestamp. – gstackoverflow Dec 25 '19 at 15:45
  • As I said: it is OK to add extra column and have it populated with a trigger. Some other processes may check the value in that column periodically and remove old rates if that. Otherwise it is not a metadata store responsibility to make assumptions combining value and something else. – Artem Bilan Dec 25 '19 at 15:48
  • 1. To add extra column do I have to remove **integration.jdbc.initialize-schema:always** ? – gstackoverflow Dec 25 '19 at 16:02
  • That’s right. Because you can’t modify scripts in the jar. Well, sounds like a separate SO thread, not comments chat – Artem Bilan Dec 25 '19 at 16:03
  • 2. How to populate extra column? metadataStore has only key and value in the interface – gstackoverflow Dec 25 '19 at 16:03
  • Let me several minutes to prepare topic – gstackoverflow Dec 25 '19 at 16:03
  • Well, wrong order for question: it uses only two columns. Everything extra can be managed externally. As I said: LAST_UPDATE can be populated with an on_update trigger on the table. All of that had to be configured on DB already – Artem Bilan Dec 25 '19 at 16:11
  • Looks like I found scenario when this approach won't work. 1. We receive new message, process it, send ack and in ack callback we fill in additional field FINISH_DATE with current timestamp 2. We receive duplicated message after callback (from the previous step) was called but before we fill in additional field FINISH_DATE. At this case app will overwrite record in the idempotency store and this message will stuck forever in the idempotency store. – gstackoverflow Dec 26 '19 at 09:18