2

I have an code which reads lots of data from a file and writes that to an excel file. The problem im facing is, when the data goes beyond the limit of heap size, its throwing an out of memory exception. I tried increasing the heap size and the program ran normally. But the problem is, there is limited RAM on my machine and if I dedicate huge space to the heap, the machine is becoming very slow. So, is there any way to free the memory after the processing some limit of data so that I need not increase my Heap size for running my code? Im relatively new to this kind of stuff, so please suggest some ideas

Surya Chandra
  • 1,561
  • 5
  • 27
  • 42
  • Show us (a small, self contained etc) version of your code where you read and write your data (only that, not all the other stuff you might need, make it as small as possible, but complete enough to show the behaviour) and we might comment on that. – Nanne May 04 '12 at 06:58
  • http://stackoverflow.com/questions/2356137/read-large-files-in-java take a look this problem. – Rangi Lin May 04 '12 at 06:59
  • I have to read data from a number of files and write that into an excel file at a single go. – Surya Chandra May 04 '12 at 07:06

3 Answers3

2

In cases like this you need to restructure your code, so that it works with small chunks of data. Create a small buffer, read the data into it, process it and write it to the Excel file. Then continue with the next iteration, reading into the same buffer.

Of course the Excel library you are using needs to be able to work like this and shouldn't requiring writing the whole file in a single go.

kgiannakakis
  • 103,016
  • 27
  • 158
  • 194
0

I think, JXL API provides such kind of limited buffer functionality.It's an open source API. http://jexcelapi.sourceforge.net/

Venkata Rammohan CH
  • 1
  • 1
  • 1
0

You can use DirectByteBuffer with Buffer.allocateDirect(byteSize); or MemoryMappedFile, they use memory space out of heap memory.

UsTa
  • 329
  • 2
  • 5
  • 10
  • However, allocating a direct buffer can also result in an OutOfMemoryError, if you hit the memory limits imposed by the OS / architecture / hardware. And this is likely to happen in this case, because the OP says his machine is short of RAM. – Stephen C May 04 '12 at 07:16
  • So, you can persist or save your data to persistent storage (HDD), maybe using Gson to save your data to disk – UsTa May 04 '12 at 10:57