0

I have a ListItemReader, which makes a repo call to read data from database. How can I partition when my primary key column in SQL results does not have consecutive values in count of 1000s (ex. 3785, 3854, 3899, 3942, 3956)?

I used SimplePartitioner but all threads read the same data. I have the @StepScope annotation in place. Also tried range partitioner but could not partition correctly due to non consecutive ids.

I expect to partition the repo results so it divides 1000s of ids equally on multiple threads (grid size provided)

avariant
  • 2,234
  • 5
  • 25
  • 33
P. Sh
  • 1
  • 1
  • The question is more generic than about a Spring Batch partitioner: how to partition a data set when there is no sequential column in it? Have you tried to `order by` your non consecutive primary key then use a paging technique (like for [oracle](https://stackoverflow.com/questions/241622/paging-with-oracle) or [mysql](https://stackoverflow.com/questions/3799193/mysql-data-best-way-to-implement-paging) to calculate pages boundaries? Each thread would then also `order by` the same column and select only the rows of its partition (using `rowid` for oracle, or `offset/limit` for mysql, etc). – Mahmoud Ben Hassine May 08 '19 at 08:04
  • Tried it, only 1 thread picks reads, while the other threads do not read any data – P. Sh May 08 '19 at 12:30

0 Answers0