0

I have a search function in which i am returning all the records of the entities to a user on searh button , now matter how much the records are(right now, it is able to search 50,000 records). now i am trying to download all these records in a csv .if the records are less, then its working fine , but when its more than 30,000, it is throwing

Edited:-

Solution:- used these lines of code 

            InputStream in = new ByteArrayInputStream(buffer.toString().getBytes("UTF-8"));
            ServletOutputStream out = response.getOutputStream();

           byte[] outputByte = new byte[4096];

             while(in.read(outputByte, 0, 4096) != -1)
                {
                   out.write(outputByte, 0, 4096);
                  }
         in.close();
            out.flush();
              out.close();*/
rcky
  • 267
  • 2
  • 4
  • 15
  • possible duplicate of [Exception in thread "main" java.lang.OutOfMemoryError: Java heap space](http://stackoverflow.com/questions/2381849/exception-in-thread-main-java-lang-outofmemoryerror-java-heap-space) – Brian Roach Mar 22 '12 at 06:32
  • When you use all the memory, you get that error. – Brian Roach Mar 22 '12 at 06:32

2 Answers2

3

Instead of writing everything to a giant in memory buffer then making a giant in memory string copy of it, get the Writer from your HttpServletResponse and write the CSV directly to the client as you create it. This way you can flush the data down the network to the client and not have to keep two entire copies of it in RAM before sending the whole thing.

Alternately of course, make the heap bigger!

Affe
  • 47,174
  • 11
  • 83
  • 83
  • just change the signature of generateCSV to also take in a Writer, pass it in, and then it implements Appendable so you really shouldn't have to change much to replace the buffer with it, other than to flush at the end instead of returning toString(); – Affe Mar 22 '12 at 06:51
  • @Affe...getting this kind of rror java.lang.OutOfMemoryError: Java heap space java.util.Arrays.copyOf(Unknown Source) java.lang.StringValue.from(Unknown Source) java.lang.String.(Unknown Source) i made an object of HttpServletResponse and using that object i got writer and then appended the whole thing....and then flushed it – rcky Mar 22 '12 at 07:34
  • Remove the StringBuffer entirely, change getCSVContentForBatch to take an Appendable, not a string buffer, and write out each pass of the loop to the client as you generate it. – Affe Mar 22 '12 at 07:39
  • @Affe.....edited the way u said it...still getting the same exception....tell me where i m doing wrong – rcky Mar 22 '12 at 07:56
  • hmm, looking at your stack trace more carefully com.ibm.db2.jcc.c.fc.getDB2XmlString(fc.java:187) might indicate your query results don't fit in memory and it never gets as far as generating CSV. What is the first part of your code referenced if you go down the rest of the stack trace? You may need to batch your query results in addition. – Affe Mar 22 '12 at 08:04
1

You should consider streaming the records over the socket rather than trying to buffer them all in memory. That would probably require you to pass in the OutputStream to your code that generates the csv.

Either that or run your server with more memory, but that's really not a good answer, because you're just putting off the OOM exception for another day.

BillRobertson42
  • 12,602
  • 4
  • 40
  • 57