(Bad form, I know, but answer my own question to document it for other people)
The updating of a counter ...
public static byte[] update_iv(byte iv[], long blocks) {
ByteBuffer buf = ByteBuffer.wrap(iv);
buf.order(ByteOrder.BIG_ENDIAN);
long tblocks = buf.getLong(8);
tblocks += blocks;
buf.putLong(8, tblocks);
return buf.array();
}
Explanation of AES/CTR-BE
This is the basic idea. If you read erickson's answer to my question you will see that the IV is basically:
<8 bytes nonce><8 bytes counter>
The counter is stored in BIG_ENDIAN format, so that if you were to pull out the counter at state 1 you'd get this:
0x0 0x0 0x0 0x1
Then when it gets to the second block it updates it to
0x0 0x0 0x0 0x2
and so forth, it can technically overflow into the nonce, but it is not suggested to encrypt that much data in the first place.
Now personally I create the nonce/counter randomly. So that it becomes even harder to guess, this is not a requirement.
What the above does is allow you to update the counter with how many blocks into the counter you want to go, it doesn't matter whether you start at 0x1 or any other counter value (random like myself).
Now, if we end on half a block or less we need to make sure we move forward in the AES-CTR for a couple of bytes, so we can simply do:
c.update(new byte[count])
where count is the amount of characters that is the distance into the block.
My implementation explained
The way I have my keyfile stored on disk (in plaintext, PLEASE DO NOT DO THIS!) is as follows:
<16 bytes AES key>
<8 bytes nonce>
<8 bytes counter>
<8 bytes (long) block count>
<4 byte partial block count>
This gives us all the information we need to append something to the end of an already encrypted file without having to first decrypt any content. Which is absolutely fantastic for log files that need to be encrypted, as well any other content that can be streamed.
Testing
The way I tested that this actually worked is as follows:
echo "1234567890ABCDEF" > file1
echo "0987654321ABCDEFGHIJKLMNOPQRSTUVWXYZ" > file2
cat file1 file2 > file3
Now, if we encrypt file1 and then append file2 we should get the same output as when we encrypt file3 so long as we use the same key/IV for both.
javac AESTest.java # Compile the java file
java AESTest key file1 append.aes
java AESTest key file2 append.aes append
Adding append tells the program to go into append mode and move the block count forward and go partially into the next CTR cycle using the aforementioned c.update() method. From there on it starts encrypting like any other time, and simply appends the data to the output file.
java AESTest key file3 noappend.aes
Since my program will simply ignore the block count/partial block count unless you pass in the argument append this will simply start encrypting the file using the same key/IV as before.
Now if we look at both files using a HexEditor or vbindiff we can verify that the two files are exactly the same, yet one had content appended to it after the fact.
Full source ...
(Please do note that this is the first time I programmed in Java since high school, which was a few years ago, please excuse the horrible code)
Full source code for my program where all of this is implemented.
import java.util.Random;
import java.security.*;
import javax.crypto.*;
import javax.crypto.spec.*;
import java.lang.String;
import java.io.File;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
public class AESTest {
public static byte[] update_iv(byte iv[], long blocks) {
ByteBuffer buf = ByteBuffer.wrap(iv);
buf.order(ByteOrder.BIG_ENDIAN);
long tblocks = buf.getLong(8);
tblocks += blocks;
buf.putLong(8, tblocks);
return buf.array();
}
public static void main(String args[]) throws Exception {
if (args.length < 3) {
System.out.println("Not enough parameters:");
System.out.println("keyfile input output [append]");
return;
}
File keyfile = new File(args[0]);
DataInputStream key_in;
DataOutputStream key_out;
DataInputStream input = new DataInputStream(new FileInputStream(args[1]));
DataOutputStream output = null;
byte key[] = new byte[16 + 16];
byte aeskey[] = new byte[16];
byte iv[] = new byte[16];
byte ivOrig[] = new byte[16];
long blocks = 0;
int count = 0;
if (!keyfile.isFile()) {
System.out.println("Creating new key");
Random ranGen = new SecureRandom();
ranGen.nextBytes(aeskey);
ranGen.nextBytes(iv);
iv = update_iv(iv, 0);
System.arraycopy(iv, 0, ivOrig, 0, 16);
} else {
System.out.println("Using existing key...");
key_in = new DataInputStream(new FileInputStream(keyfile));
try {
for (int i = 0; i < key.length; i++)
key[i] = key_in.readByte();
} catch (EOFException e) {
}
System.arraycopy(key, 0, aeskey, 0, 16);
System.arraycopy(key, 16, iv, 0, 16);
System.arraycopy(key, 16, ivOrig, 0, 16);
if (args.length == 4) {
if (args[3].compareTo("append") == 0) {
blocks = key_in.readLong();
count = key_in.readInt();
System.out.println("Moving IV " + blocks + " forward");
iv = update_iv(iv, blocks);
output = new DataOutputStream(new FileOutputStream(args[2], true)); // Open file in append mode
}
}
}
if (output == null)
output = new DataOutputStream(new FileOutputStream(args[2])); // Open file at the beginnging
key_out = new DataOutputStream(new FileOutputStream(keyfile));
Cipher c = Cipher.getInstance("AES/CTR/NoPadding");
SecretKeySpec keySpec = new SecretKeySpec(aeskey, "AES");
IvParameterSpec ivSpec = new IvParameterSpec(iv);
c.init(Cipher.ENCRYPT_MODE, keySpec, ivSpec);
if (count != 0) {
c.update(new byte[count]);
}
byte cc[] = new byte[1];
try {
while (true) {
cc[0] = input.readByte();
cc = c.update(cc);
output.writeByte(cc[0]);
if (count == 15) {
blocks++;
count = 0;
} else {
count++;
}
}
} catch (EOFException e) {
}
cc = c.doFinal();
if (cc.length != 0)
output.writeByte(cc[0]);
// Before we quit, lets write our AES key, start IV, and current IV to disk
for (int i = 0; i < aeskey.length; i++)
key_out.writeByte(aeskey[i]);
for (int i = 0; i < ivOrig.length; i++)
key_out.writeByte(ivOrig[i]);
System.out.println("Blocks: " + blocks);
System.out.println("Extra: " + count);
key_out.writeLong(blocks);
key_out.writeInt(count);
}
}
