Spring Batch : Compare Data Between Database

Question

I have two databases, Oracle and MySQL. The target is saving value from a table in Oracle into MySQL, with the requirement: data does not exist in MySQL table.

But I had trouble understanding spring batch. In step, it contains itemReader, itemProcessor, and itemWriter. If my understanding is correct, a step only can have single itemReader (cmiiw)

My plan is to make 2 steps:

Step 1. List from MySQL table

Step 2. Compare T from Oracle with List (step 1) using a primary key ex: id.

If the approach is correct, My question is: how make a List in step? and how passing it into step 2?

I kindly need guidance in this, thank you

Ashish Shetkar · Accepted Answer · 2019-02-20T07:20:15.270

0

for your requirement , you can use a custom Item reader , for example

public class MyItemReader implements ItemReader<MyDTO> {

    @Override
    public MyDTO read() throws Exception {
        MyDTO myDto = null;

        /*
        your custom code 

        */

        return myDto;
    }
}

But the better approach as per the Sprig batch structure will be

So you can handle your logic in processor , in the processor you can compare data with oracle ,

Step 1. List from MySQL table - will be handled in spring batch reader

Step 2. Compare T from Oracle with List (step 1) using a primary key ex: id. - will be handled in processor

to increase performance , i would suggest you to make use of some cache if oracle data is not too much. If oracle data is large then you can just go with the simple way of looking out for data on oracle DB in your processor

edited Feb 20 '19 at 07:20

answered Feb 18 '19 at 14:30

Ashish Shetkar

1,414
2
18
35

thank you so much, i use your suggesstion "looking out for data on oracle DB in your processor" and it works – relesa telco Feb 20 '19 at 02:35
great , in case if you are facing performance issues or you want the better performance , try and look out for cache implementation for the Oracle DB lookup - there are so many open source cache implementation are available in market . For ex - you can check LRU cache , follow this url - https://stackoverflow.com/questions/224868/easy-simple-to-use-lru-cache-in-java – Ashish Shetkar Feb 20 '19 at 06:32
1

Bro, i still confused for making itemReader return as list. For now my itemReader still returning a single obj, can you give a example link? Thank you – relesa telco Feb 22 '19 at 04:49
1

Hi , actually reader can never return the list , follow the docs https://docs.spring.io/spring-batch/trunk/reference/html/readersAndWriters.html , reader and processor will aleays have one single object input and one object as return , however - you can write multiple object at once - for this you need implement your own chunk - you can give chunk size - try searching for chunk in spring batch – Ashish Shetkar Feb 22 '19 at 06:39
1

So itemReader and ItemProccessor only accept a single obj? If i want make comparation of list from source databases (created in itemReader, passing into itemProcessor) and list from target (jdbc template in itemProcessor), it not possible? – relesa telco Feb 22 '19 at 07:24
1

you can do one thing , implement a itemWriterListener - follow https://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/core/ItemWriteListener.html and then use the method beforeWrite for this - you will be able to compare the lists – Ashish Shetkar Feb 22 '19 at 09:12

Spring Batch : Compare Data Between Database

1 Answers1

Linked