0

I am currently making my own REST API service but I stumbled into a problem. I have a method, which exports an excel file to user:

@Override
public File exportMappings(Long from, Pageable pageable) throws IOException {
    File tempFile;
    try (AbstractMarshaller marshaller = getMarshaller()) {
        Page<Sale> sales = saleRepository.findByIdGreaterThan(from, pageable);
        marshaller.marshallSheet(ExportSaleMappingContentResolverImpl.TAB_NAME, saleMappingContentResolver,
                sales.iterator());
        marshaller.finalizeWritings();
        tempFile = marshaller.getFile();
    }
    return tempFile;
}

The problem is that if I want to export millions of records in my excel file, Page<Sale> sales = saleRepository.findByIdGreaterThan(from, pageable); will be overloaded with elements and my service will crash because of OutOfMemoryError. I understand, that I need to create my own Iterator which will be able to divide my query in pieces so page won't be overloaded. But I don't know how to do it.

XDePi
  • 1
  • I guess you are using apache poi to create excel file. If so, it's common problem. This lib doesn't support streaming and consums really great amount of memory. – Boris Chistov May 27 '21 at 10:49
  • @BorisChistov , yes you are right. But I think there might be solution to this problem anyways, I am just not that qualified to be able to solve this mystery – XDePi May 27 '21 at 10:58
  • try to export it to csv in streaming way, or just iterate over all results log then for example, so you could understand if the problem in poi or in repository – Boris Chistov May 28 '21 at 11:13

1 Answers1

0

are you starting your JVM with default memory ? One option is to raise your -xmx parameter to like 2GB. That may solve the problem temporarily but for a permanent solution you will need a more efficient process. Now I did find this entry in stackoverflow for row iterator with Apache POI : How to use rowiterator in apache poi with java?

here is another tutorial on POI with iterator also : https://www.tutorialspoint.com/apache_poi/pdf/apache_poi_spreadsheets.pdf

AamirAli
  • 11
  • 2