2

I need to read a file into an array of Bytes.The entire file needs to be read into the array. The problem is I am getting an OutOfMemory Error since the file size is too large. Increasing -XmX does not seem to have any effect. Here is the code snippet :

InputStream in = new FileInputStream(file);
long length = file.length();                        
byte[] out = new byte[(int)length]; 
// Process the byte array

The problem occurs during the byte array instantion. Is there a less memory intensive workaround to this issue?

Goutham
  • 2,759
  • 9
  • 30
  • 36

4 Answers4

3

You need to have far more free memory than the largest file you can have to use this approach. Given a machine with 24 GB costs less than £2K, this isn't as silly idea as it used to be. Actually the 2GB limit for a byte[] is more of a headache in some situations.

However, the usual way to read an InputStream is to read a block of say 8KB at a time. This way you only need to have far more than 8KB free.

BTW: You can use -mx1g instead of the option you are using.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • why doesn't Java just allocate virtual memory via the operating system? – simpleuser Feb 25 '16 at 20:12
  • @user1663987 it does, but it has managed memory and the maximum size of the managed memory is limited. All other regions, direct memory, metaspace, thread stacks, shared libraries are all created via the OS. – Peter Lawrey Feb 26 '16 at 08:47
2

No, if your file is too large to fit into memory, it's too large to fit in memory.

A better solution would be to try to process the stream as a stream, rather than loading the whole thing into memory. Without knowing what processing you're trying to achieve, we can't really tell whether or not that's feasible.

For example, if you're just trying to compute a secure hash of the file, you should be able to do that without loading significant amounts of data in at a time - but if your processing requires random access to data, you may need to use RandomAccessFile instead.

Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194
1

The workarround would be, not to load the entire file into the RAM. In fact you can't do this for large files, because you have to allocate lots of memory at one peace, which may not work.

The question ist: Do you really need the entire file in memory?

edit

InputStream in = new FileInputStream(file);
long length = file.length();     
// At this point a warning should appear, because the code would
// not work for files larger than Integer.MAX_VALUE                   
byte[] out = new byte[(int)length];
Christian Kuetbach
  • 15,850
  • 5
  • 43
  • 79
  • Since the problem occurred in an existing code, I was wondering if I could solve the issue without changing the processing part. Looks like I will have to read chunks of the file at a time. Thanks! – Goutham Jan 28 '11 at 09:58
0

How about using a memory mapped file: FileChannel

From http://www.java-tips.org/java-se-tips/java.nio/how-to-create-a-memory-mapped-file-3.html:

try {
    File file = new File("filename");

    // Create a read-only memory-mapped file
    FileChannel roChannel = 
      new RandomAccessFile(file, "r").getChannel();

    ByteBuffer readonlybuffer = 
      roChannel.map(FileChannel.MapMode.READ_ONLY, 
0, (int)roChannel.size());

    // Create a read-write memory-mapped file
    FileChannel rwChannel = 
      new RandomAccessFile(file, "rw").getChannel();

    ByteBuffer writeonlybuffer= 
      rwChannel.map(FileChannel.MapMode.READ_WRITE, 
  0, (int)rwChannel.size());

    // Create a private (copy-on-write) memory-mapped file.
    // Any write to this channel results in a private 
    // copy of the data.
    FileChannel pvChannel = 
      new RandomAccessFile(file, "rw").getChannel();

    ByteBuffer privatebuffer = 
      roChannel.map(FileChannel.MapMode.READ_WRITE, 
  0, (int)rwChannel.size());

} catch (IOException e) {
}
SimonC
  • 6,590
  • 1
  • 23
  • 40