0

I am using Spring batch solution for my project. I have a situation where I need to read ID's from a file and for those Ids I need to pull the data from the database, process it and then write it to an output file.

Currently I use a FlatFileItemReader to read the data from the file and then in the processor I collect them and for every thousand count (in clause has 1000 limitation) I hit the database to collect the data for those Ids and use a Writer to write the output.

But i somehow feel there should be a better way to do this. If I have access to the DB which I am hitting I could store these Ids in a temp table to join and get the output. But I cannot do that. Is there anyway to connect the readers.

Kenneth
  • 23
  • 6

1 Answers1

0

I think, you should design you job in two steps while passing output of step # 1 to step #2 OR by persisting output of step#1 to intermediate storage if that is going to simplify reader logic in step#2.

STEP#1: Consisting of a FlatFileItemReader -> Processor .

STEP#2: DB reader - > Processor -> Writer

Going by your problem description, it seems you can add a writer in STEP#1 to write to temporary/intermediate table.

Also, it looks you don't need processor in STEP#2.

STEP#1 data can be passed to STEP#2 using ExecutionContext, See this and related question

Also, sentences like , But I cannot do that aren't helpful unless you add more explanation about why?

Community
  • 1
  • 1
Sabir Khan
  • 9,826
  • 7
  • 45
  • 98
  • Thanks for your reply. "But I cannot do that" means I have no write access to that database or I am not supposed to create any objects in the DB since it is not within our project limits. So any other solution u suggest rather than creating an interim table. – Kenneth Sep 08 '16 at 13:20
  • so what is the issue? I guess , you need to directly pass data from step 1 to step 2 without persisting it. I have not implemented that logic but guess that passing data is possible as explained in those links. – Sabir Khan Sep 09 '16 at 06:34