0

I have a huge amount of data (~4 GB) returned from db as an iterable list. I want to return this data from my rest api as a JSON. I can only use 512 mb of Java heap for this task. What approach I can take for this? I have tried Java stream but as of now, I always hit heap out of space error. Any example will be of great help. ☺️

ReactivePsycho
  • 19
  • 1
  • 4
  • 4
    Paginate the results. Nobody ever needs to load 4GB of results at the same time. If it is not displayed data but rather files, then use another solution, like storing them and just returning the URL to the file. – AntoineB Aug 09 '18 at 09:26
  • This might help. https://stackoverflow.com/questions/9390368/java-best-approach-to-parse-huge-extra-large-json-file/9390517 – codebot Aug 09 '18 at 09:34
  • It cannot be a file. It's like hit the get method from browser and the JSON gets slowly loaded to browser. I actually tried to convert the iterable to Java 8 streams and return the stream from the API without luck. – ReactivePsycho Aug 09 '18 at 09:35
  • @ReactivePsycho what client would need 4GB of data at once? Your front-end would die processing such a large response. – Andrew Tobilko Aug 09 '18 at 09:48
  • I am not saying that I need 4gb data at once. I merely want a way to return the data lazily from the api without ever needing to load the whole data in memory. My class is like : class A { list hugeList; } . I am setting the hugeList with the data I am getting from database and trying to return the object of class A from my API. – ReactivePsycho Aug 09 '18 at 09:57

1 Answers1

2

I propose this solution :

  1. Add a LIMIT on your query, to get only a part of your data (< 512mb)
  2. Process the partial data you got and return it to the client with a flag "uncomplete".
  3. Ask the server the following part
  4. Repeat step 1 to 3 until there's no data to process
  5. Return the last part with the flag "complete"
  6. Profit
S. Miranda
  • 140
  • 1
  • 1
  • 9