1

I have a Java application where we use spring data JPA to query our oracle database. For one use case, I need to fetch all the records present in the table. Now the table has record count of 400,000 thousand and it might grow in the near future. I don't feel comfortable pulling all records into the JVM since we don't know how large they can be. So, I want to configure the code to fetch specific number of records at a time say 50,000 and process before it goes to next 50,000. Is there a way I can achieve this with JPA? I came across this JDBC property that can be used with hibernate hibernate.jdbc.fetch_size. What I am trying to understand is if I use repository.findAll() returning List<Entity>How can a fetch Size work in this case? because List will have all the entities. I was also looking into repository methods returning Stream<>, not sure if I have to use that. Please do suggest. If there can be better solution for this use case?

Thanks

user16353001
  • 49
  • 8
  • I don't think fetch size has benefits if you need to fetch everything into a static collection like a list, https://stackoverflow.com/a/48129503/995891 - even if it would do multiple fetch operations you'd still load all data into memory at once. How streaming and fetch size behaves seems to depends on more than just hibernate: https://stackoverflow.com/q/55952655/995891 – zapl Oct 18 '22 at 18:40
  • you can get only part of data with pagination, but question is what you want to do with each portion of data? Try to describe – notAPPP Oct 18 '22 at 19:10
  • @notAPPP I just need to execute small logic on each db record and save it in-memory. – user16353001 Oct 18 '22 at 20:28

1 Answers1

0

With JPA you can use the Pagination feature, means you tell the Repository how many result should be present at one page. (E.g. 50 000)

For more information follow up here https://www.baeldung.com/jpa-pagination

mhdl
  • 7
  • 3