1

I have a Spring Batch application with two databases: one SQL DB for the Spring Batch meta data, and another which is a MongoDB where all the business data is stored. The relation DB still uses DataSourceTransactionManager. However I dont think the Mongo writes are done within an active transaction with rollbacks. Here is the excerpt from the official Spring Batch documentation on MongoItemWriter:

A ItemWriter implementation that writes to a MongoDB store using an implementation of Spring Data's MongoOperations. Since MongoDB is not a transactional store, a best effort is made to persist written data at the last moment, yet still honor job status contracts. No attempt to roll back is made if an error occurs during writing.

However this is not the case any more; MongoDB introduced ACID transactions in version 4. How do I go about adding transactions to my writes? I could use @Transactional on my service methods when I use ItemWriterAdapter. But still dont know what to do with MongoItemWriter... What is the right configuration here? Thank you.

BlackLog
  • 301
  • 1
  • 5
  • 17

1 Answers1

4

I have a Spring Batch application with two databases: one SQL DB for the Spring Batch meta data, and another which is a MongoDB where all the business data is stored.

I invite you to take a look at the following posts to understand the implications of this design choice:

In your case, you have a distributed transaction across two data sources:

  • SQL datasource for the job repository, which is managed by a DataSourceTransactionManager
  • MongoDB for your step (using the MongoItemWriter), which is managed by a MongoTransactionManager

If you want technical meta-data and business data to be committed/rolled back in the scope of the same distributed transaction, you need to use a JtaTransactionManager that coordinates the DataSourceTransactionManager and MongoTransactionManager. You can find some resources about the matter here: https://stackoverflow.com/a/56547839/5019386.

BTW, there is a feature request to use MongoDB as a job repository in Spring Batch: https://github.com/spring-projects/spring-batch/issues/877. When this is implemented, you could store both business data and technical meta-data in the same datasource (so no need for a distributed transaction anymore) and you would be able to use the same MongoTransactionManager for both the job repository and your step.

Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
  • Thanks for taking the time, @Mahmoud. Do you happen to have a rough estimate on the ETA of this feature? Thanks. – BlackLog Mar 03 '21 at 19:19
  • No, but we might consider it for the next major release v5 (for which we did not set a date yet). But in the meantime, you should be able to achieve what you are looking for with a JTA transaction manager. – Mahmoud Ben Hassine Mar 03 '21 at 20:13