What is the proper way to handle large files?

Question

How should a large file be handled in java when you need to run the bytes through a variety of methods?

The way I was doing it before is like this:

private byte[] inputStreamToByteArray(InputStream inputStream) {
   BufferedInputStream bis = BufferedInputStream(inputStream);
   ByteArrayOutputStream baos = new ByteArrayOutputStream();

   byte[] buffer = new byte[8192];

   int nRead;
   while((nRead = bis.read(buffer)) != -1) {
      baos.write(buffer, 0, nRead);
   } 

   return baos.toByteArray();
}

I get a java out of memory error doing it this way because my byte array gets too large.

So I tried stringing together streams, but I'm not sure if that is the proper way to do it because I don't understand enough about streams.

Should large files be handled using chunks from a byte array or by passing around inputstreams?

score 3 · Accepted Answer · answered Jul 24 '13 at 22:34

3

Should large files be handled using chunks from a byte array or by passing around inputstreams?

Large files should be read from a file in chunk of say 8192 bytes, exactly as you do in the example. Instead of copying the data to an array and processing the array, just process the data as you read it.

answered Jul 24 '13 at 22:34

Peter Lawrey

525,659
79
751
1,130

Thank you! That is where I get confused. How exactly do I process 8192 bytes in another method and then return to get the remaining bytes until they are all processed? – nullByteMe Jul 24 '13 at 22:38
Have the other method read what it need to read from the InputStream or BufferedReader for lines of text. No need to jump back and forth between two methods. – Peter Lawrey Jul 24 '13 at 22:39

score 1 · Answer 2 · edited May 23 '17 at 11:46

Either:

Process file via Memory Mapped Files. Handles at least up to 2GB size - if you want to dedicate that much memory! Integrates with operating system native IO threads and memory buffers to increase performance a little.
```
 java.nio.MappedByteBuffer buff = file.getChannel.map();
```
Then access various parts of the buffer - they will be paged into java memory in turn, so still some IO chunking occuring. But logically, to your program it looks as if it's processing entire file (with some abstraction leakage in I/O performance during paging).
Process chunks as you read them - instead of appending to an ever-growing ByteArrayOutputStream. In order to process chunks: read large enough chunks so they have meaning to your program. OR aggregate the pieces only to the point that they have meaning and can be processed and discard them before the next read.

Often (2) performs well, but (1) can perform well and is occasionally simpler but is more memory-expensive.

See also: Most Robust way of reading a file or stream using Java (To prevent DoS attacks)

:)

What is the proper way to handle large files?

2 Answers2