How to read a file as byte[] in Java when it is bigger than your RAM?

Question

I have written a Java program which encrypts a file using my custom algorithm. It reads a file as byte[] and the my function modifies this array. Finally, this array is saved as encrypted file. But the problem is that if I read a file which is greater than my RAM, it no longer can encrypt. So what's the solution?

Edit: For now assume that there is no encryption. I just have to reverse the file.

Encrypt the file in chunks of n bytes, using a stream to limit what you're holding in memory at any one time. Decryption will need to work on a chunked basis too. — Steve Chaloner, May 05 '17 at 08:21
What kind of encryption algorithm do you use? Is it a `Block Cipher` or `Stream Cipher`? — Sanket Makani, May 05 '17 at 08:28
My algorithm involves permutation of bytes and I don't want to permute bytes among a smaller chunk because then a statistical attack can detect the pattern of shuffling in bigger files. — hhj8i, May 05 '17 at 08:28
Add the requirements of the encryption into your question, did you need every bytes to start the encryption. — AxelH, May 05 '17 at 08:31
You can use a RandomAccessFile, then read the input file and write the bytes wherever you like in the target file. Then you never need to read all of the input bytes at once. — matt, May 05 '17 at 08:39
if your cryptography depends heavily on permutation of bytes, and if the chunks were small, it could be attacked, then it is probably not strong solution. What about very small (single word or even single byte) files ? Also how big files have you tried OR what size are you aiming for? — diginoise, May 05 '17 at 08:43
If you're inventing your own cryptographic algorithm, you've got bigger problems than running out of memory. — Kayaman, May 05 '17 at 08:49
@hhj8i you probably simplied to much ;) to reverse a file you could simply read the file char by char and prepend a tmp file with each char. — AxelH, May 05 '17 at 10:00
@AxelH But file is binary data and not just strings.Like I have an executable to process ;) — hhj8i, May 05 '17 at 10:03
@hhj8j Ok, read each byte one by one and prepend the tmp file. Same idea. — AxelH, May 05 '17 at 10:04

jropella · Answer 1 · 2017-05-05T10:58:02.907

2

Instead of reading the entire thing to a buffer and then writing it all out at once in encrypted form, you can use streams to read and write chunks at a time. Specifically, you could use a CipherOutputStream to encrypt as you go.

Just as an example of the kind of thing you might do:

byte[] buffer = new byte[4096];
FileInputStream fileInStream = new FileInputStream("in.txt");
FileOutputStream fileStream = new FileOutputStream("test.bin");
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
CipherOutputStream cipherStream = new CipherOutputStream(fileStream, cipher);

while(fileInStream.read(buffer) > 0){
    cipherStream.write(buffer);
}

If you are just trying to reverse the entire file without reading the whole thing into memory all at once, you could do something like this, noting that you'll need to reference Commons.Lang in order to get the ArrayUtils.reverse functionality:

byte[] buffer = new byte[4096];
File file = new File("in.txt");
FileInputStream fileInput = new FileInputStream(file);
FileOutputStream fileOutput = new FileOutputStream("out.bin");

int index = (int)(file.length() - 4096);
int bytesRead = -1;
while((bytesRead = fileInput.read(buffer, index, 4096)) > 0 && index >= 0){
    index = Math.max(index - 4096, 0);
    if(bytesRead < 4096){
        buffer = Arrays.copyOf(buffer, bytesRead);
    }
    ArrayUtils.reverse(buffer);
    fileOutput.write(buffer);
}

edited May 05 '17 at 10:58

answered May 05 '17 at 08:22

jropella

589
5
14

Let's assume that instead of encryption, I have to reverse the byte array. So will using stream be a way to go? – hhj8i May 05 '17 at 08:25
In that case you might consider implementing your own extension of a stream reader/writer which simply reverses the array of bytes. This way you're still dealing with chunks from the consuming side. – jropella May 05 '17 at 08:26
@jropella reversing only bytes within chunks rather than entire file... So if file was `12345678` and chunk was of size 2, then you would get `21436587`. Reversing entire file would require reading the original and writing to new file if space permits, otherwise you are looking at random access file and reversing byte at the time (it will cost time). – diginoise May 05 '17 at 08:35
1

@diginoise: what about reading it backwards? Start at index = size, read from index to index - x until index = 0, reversing as you go. – jropella May 05 '17 at 08:36
@jropella exactly. as I mentioned to do it 'in place' it will take long time (maybe not prohibitively long) compared to reading original and writing to new file. – diginoise May 05 '17 at 08:40
@jropella Whether the file is read backwards or forwards, the same amount of RAM is needed – hhj8i May 05 '17 at 09:54
yes, but if you read chunks of bytes at a time you don't need as much RAM. @diginoise was simply raising the point that if you merely reverse individual chunks at a time you could potentially run into problems interpreting it later on., – jropella May 05 '17 at 10:28

How to read a file as byte[] in Java when it is bigger than your RAM?

1 Answers1