0

I have below use case where we have to do the following using spring batch.

  1. Read big file
  2. process each records/line from file
  3. write it to DB

Currently it is single thread and working fine but i want to make it multithread where multiple thread will process and write to DB.

I have following approaches .

  1. if I use itemReader, itemProcessor, itemWrite then reading file using multiple thread is not a good idea and might degrade the performance. where multiple thread will reading same file using itemReaderClass.
  2. Second solution is , in 'step1' i read file and place in BlockingQueue and in 'step2' would
    be multithread where i read,process,write from BlockingQueu.Then run 'step1' and 'step2' in parallel. so 'step1' is single threaded producer and 'step2' is multi threaded consumer.

Please suggest which one is right way to do it.

kamalp
  • 11
  • 3
  • 2nd approach check https://stackoverflow.com/questions/18999724/spring-batch-one-reader-multiple-processors-and-writers?rq=1 – Abhishek Feb 19 '20 at 09:18
  • For approach 1, you can use a [Multi-threaded step](https://docs.spring.io/spring-batch/docs/4.2.x/reference/html/scalability.html#multithreadedStep). `using multiple thread is not a good idea and might degrade the performance` Can you elaborate on this? Have you tried that and noticed a performance degradation? Without numbers, it is difficult to come to this conclusion. For approach 2, there is an open issue: https://github.com/spring-projects/spring-batch/issues/2044, I implemented a PoC for it here: https://github.com/benas/spring-batch-sandbox/tree/master/batch1538. HTH. – Mahmoud Ben Hassine Feb 19 '20 at 12:35
  • @MahmoudBenHassine If i go with approach 1 then multiple threads will be reading same file which is not good idea in general. we tried same in spring batch , where multiple threads read same file but all the thread were giving same line to us. – kamalp Feb 19 '20 at 17:27
  • @kamalp You can use `AsynchronousFileChannel` from NIO to read file asynchronously with different file position. – Binu Feb 20 '20 at 04:46
  • Thanks @binu But i don't have much flexibility in reading side. – kamalp Feb 22 '20 at 18:15

0 Answers0