Copying whole file to memory vs. doing multiple disk accesses

Question

I'm writing a java program where I'm encrypting and decrypting files. I'm treating the files as basically byte arrays - take N bytes, encrypt and write to new files, repeat. And similarly when decrypting.

For this, I am currently copying the entire file into a byte array:

byte[] fileAsBytes = Files.readAllBytes(Paths.get(pathToFile))

But then with larger files, I run out of heap space. (I know I can increase the available memory to the JVM, but that is only dodging the issue. Plus I also have to run this on Android phones, where memory is even more limited).

So, I'm considering to only read N bytes at a time from disk, rather than copy all bytes and then read them from memory.

My question is - what is the best way to do this? I don't need random access, since I'll be reading the files sequentially. Also, how much of a slowdown am I looking at? Because from what I've studied, hard disk access is much slower than memory access.

score 3 · Accepted Answer · answered Apr 05 '15 at 08:59

3

You don't have any choice, so 'slowdown' isn't a consideration. You have to do it in bite-sized chunks. Say 64k at a time. You may even find it's quicker.

answered Apr 05 '15 at 08:59

user207421

305,947
44
307
483

I agree, you should read/write files in chunks. Something like that is done in utils from Apache Commons IO and others, for example see here: http://stackoverflow.com/a/22128215/1028256 – Mixaz Apr 05 '15 at 09:32

score 0 · Answer 2 · answered Apr 05 '15 at 13:24

The easiest way to do this is with the CipherInputStream and CipherOutputStream classes. These classes take care of the buffering, or 'piecemeal' encrypting, aspects for you. They aren't necessarily perfect depending on what you want to do. If they are lacking then you can manage the buffering yourself. The Cipher class is certainly fully stocked with methods to encrypt/decrypt in convenient chunks.

Copying whole file to memory vs. doing multiple disk accesses

2 Answers2