Java Performance - How to write big array to disk/sdcard with high performance?

Question

Is there a way in Java to write to disk a large array of, say, integers? I am doing this on an Android, and have not found a method that comes anywhere close to native C code.

The resulting file need not be portable to different machines with different representations, so logically just a bulk write of the underlying bytes should be sufficient. But I don't know how to do that efficiently from Java.

I have tried searching the net, and tested the following:

• Serialization - very slow, as expected.
• Using NIO - still slow - Android trace reveals operations one at a time per integer:

Thanks in advance

---

NIO code:

int[] array = new array[10000000];

...

raf = new RandomAccessFile(ti.testFileName, "rw");
chan = raf.getChannel();
MappedByteBuffer out = chan.map(FileChannel.MapMode.READ_WRITE, 0, array.length*4);
ib = out.asIntBuffer();
ib.put(array);
out.force();
raf.close();

Answer 1

You said it was slow but the speed is likely to depend on speed of your disk subsystem. You should be able to write 40 MB to a regular disk in about half a second to commit to disk.

The following uses NIO and takes 665 ms to write and 62 ms on a workstation. The read and write shuffles the same amount of data around, but the read can take its data from the OS cache, the difference how long it takes to write to disk.

int[] ints = new int[10 * 1000 * 1000];
long start = System.nanoTime();

ByteBuffer byteBuffer = ByteBuffer.allocateDirect(ints.length*4+4);
byteBuffer.putInt(ints.length);
IntBuffer intBuffer = byteBuffer.asIntBuffer();
intBuffer.put(ints);
byteBuffer.position(0);

FileChannel fc = new FileOutputStream("main.dat").getChannel();
fc.write(byteBuffer);
fc.force(false);
fc.close();
long time = System.nanoTime() - start;
System.out.println("Write time " + time / 1000 / 1000 + " ms.");

long start2 = System.nanoTime();
FileChannel fc2 = new FileInputStream("main.dat").getChannel();
ByteBuffer lengthBuffer = ByteBuffer.allocate(4);
while(lengthBuffer.remaining()>0) fc2.read(lengthBuffer);
int length = lengthBuffer.getInt(0);

int[] ints2 = new int[length];
ByteBuffer buffer2 = ByteBuffer.allocateDirect(length*4);
while(buffer2.remaining()>0 && fc2.read(buffer2) > 0);
buffer2.flip();
buffer2.asIntBuffer().get(ints2);
long time2 = System.nanoTime() - start2;
System.out.println("Read time " + time2 / 1000 / 1000 + " ms.");

I have added the length to the start of the file so it doesn't have to be assumed. BTW: There was a bug in the write which I have fixed.

Answer 2

I have no idea about the Android implementation, but in standard Java, good old-fashioned IO often outperforms NIO.

For example I believe the following code should be relatively fast if you have an array of bytes:

byte[] bytes = new byte[10000];
// ...
FileOutputStream out = new FileOutputStream(...);
try {
    out.write(bytes);
} finally {
    out.close();
}

Bear in mind that this will block until the entire array of bytes is written. But you don't say whether non-blocking behaviour is a problem or not.

Another thing you don't mention is how you intend to encode the integers when writing into the file. You need to perform the encoding in memory before writing to file, but it's possible that the array is too large to encode all at once, in which case you can encode/write in blocks of several hundred K.

Answer 3

Peter,

When something seems too good to be true, it usually is. 89msecs to write 40MB of data suggests your HDD has a bandwidth of much larger 500MB/sec (since you also included the time to open and close the file). That is unlikely to be true. Did you check the file in fact is of size 40MB. Also, I would suggest that you init the buffer to see the file content are not all zeros. May be an untouched buffer is just skipped. Whatever it is, the number you have is too good to be true.

Thanks.

Answer 4

Consider looking at buffering your outputstream