Spring Batch Crud Database

Question

My problem is more or less asked here Spring Batch : Compare Data Between Database however I still cannot get my head around it. Maybe it's a bit different.

I have A datasource and I want to write into database B.

I have full trust in A datasource, so if;

1. A Does contain the record that B does not, I have to add B.
2. A Does not contain the record that B does, I have to delete from B
3. A does contain, B does contain, I check and update the record in B accordingly.

I thought my approach would be simple as;

1. Read Person from A datasource
2. Read Person from B datasource
3. (Those two Person can be having different entities)
4. Compare and find the ones to Add,Update,Delete.
5. Update the database.

However since I am pretty newbie to Spring Batch, the implementation is kind of ending up to a spaggetti code which I don't want and want to learn the right way for it.

So; I created this job below

@Bean
public Job job() {
    return jobBuilderFactory
            .get("myNewbieJob")
            .start(populateARepository())
            .next(populateBRepository())
            .next(compareAndSubmitCountryRepositoriesTasklet())
            .build();
}

To explain; populateARepository() populateARepository() : I have a Repository object just contains a list. This step just does add records to the list.

The part that I don't like is that compareAndSubmitCountryRepositoriesTasklet() is basically comparing those repositories... and then I don't know what to do.

• If I create a DB access and push from that class, I won't like it, because I just wanted it to be a step where I find the differences.
• If I create another class which contains 3 separate lists for toUpdate,toDelete,toInsert, and then in the next step somehow use that repository... that sounded wrong to me as well.

So, here I am. Any kind of guidance is appreciated. How would you deal in this situation?

Thank you in advance.

Answer 1

Before talking about Spring Batch, I would first look for an algorithm to solve this problem. If I understand correctly, you basically need to replicate the same state of records in database A into database B. What you can do is:

• Read Person items from database A
• Use an item processor to do the comparison with table B. Here, you would mark the item accordingly to be inserted, updated or deleted
• Use an item writer that checks the type of record and do the necessary operation. Here, you can create a custom writer or use a ClassifierCompositeItemWriter (see this example)

This approach works well with small/medium datasets but not for large datasets (due to the additional query for each item, but this is inherent to the algorithm itself and not the implementation with Spring Batch).