Saving large short array android

Question

I'm basically looking for a way to save and reload a few large arrays (about 5 million short) in a fastish way on Android. My app needs to save them in a way where I could get back to them much later, so I can't just hold them in memory...

So far, I've tried converting them to a byte[] array and they seem to save successfully, but I can't get the data back, this is how my saving code works (it's actually to separate functions, simplified here):

ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOuputStream oos = new ObjectOutputStream(baos);
oos.writeObject(data);
oos.close();
baos.close();
FileOutputStream fos = new FileOutputStream(filename); // valid absolute path
fos.write(baos.toByteArray());
fos.close();

And the loading part is where I get stuck, how do I get a short[] from a byte[]? I also read that a database might work too, but is it quick enough?

I've looked all around Stackoverflow and Google and can't seem to find somebody with a similar problem, or at least a solution to it, but as a beginner I might have missed something obvious...

zapl · Answer 1 · 2013-08-07T19:50:43.140

6

If data is a short[] and you want to write the whole thing into a file, don't use a buffer in memory, write it directly.

ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(filename));
try {
    oos.writeObject(data);
} finally {
    oos.close();
}

If you write it via ObjectOutputStream, you will have to read it via ObjectInputStream since it's not just writing the pure data but also some type information. If you put a short[] in, you get a short[] back (you could try to skip those bytes but you would have to analyze what the stream is actually writing). This also applies if your ObjectOutputStream writes into a ByteArrayOutputStream.

If you don't want to handle that mess, do what ObjectOutputStream basically does:

DataOutputStream dos = new DataOutputStream(new FileOutputStream(filename));
try {
    for (int i = 0; i < data.length; i++) {
        dos.writeShort(data[i]);
    }
} finally {
    dos.close();
}

DataOutputStream writes plain data so you can simply read the data directly as a byte[] if you want. If byte order matters: it's using Big-Endian.

Since this approach writes single bytes instead of chucks of byte[] it benefits from using a BufferedOutputStream in between. The default uses an 8kB buffer, but it can increased which could give even better results. Writing the data will now convert short into bytes in memory and once enough data is available the whole chunk will get pushed to low level OS functions to write it.

int bufferSize = 32 * 1024;
DataOutputStream dos = new DataOutputStream(
        new BufferedOutputStream(
                new FileOutputStream(filename),
                bufferSize)
        );
try {
    for (int i = 0; i < data.length; i++) {
        dos.writeShort(data[i]);
    }
} finally {
    dos.close();
}

edited Aug 07 '13 at 19:50

answered Aug 07 '13 at 17:58

zapl

63,179
10
123
154

Using your and @dst's answers it seems to work, at least for output, but it takes about 13 seconds? Is there any reason it's so long? – NiBr Aug 07 '13 at 18:15
Try putting a `BufferedOutputStream` between file and data stream. You might be writing single elements which is painfully slow. – zapl Aug 07 '13 at 18:19
I was using your first code, it still takes 13 secs with the buffer. I'll try the second one with the buffer and see if that fixes it. – NiBr Aug 07 '13 at 18:24
If I give up on flexibility by using the DataOutputStream, it takes 11 seconds, so I guess I'll stick with the original OOS. But is it normal for ~5 million shorts to take that time to write on a phone? – NiBr Aug 07 '13 at 18:32
@NiBr isn't `new DataOutputStream(new BufferedOutputStream(new FileOutputStream(filename)))` helping? value conversion is unfortunately pretty slow in Java since it's actually shifting and bitmanipulating each thing. Lower lever languages can simply interpret memory in the desired way and iterate byte-wise over a short array. – zapl Aug 07 '13 at 18:42
oops, didn't see that it didn't help – zapl Aug 07 '13 at 18:44
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/35020/discussion-between-nibr-and-zapl) – NiBr Aug 07 '13 at 18:51

score 1 · Answer 2 · edited May 23 '17 at 12:20

Use the file streams directly without buffering through an ByteArrayOutputStream / ByteArrayInputStream. The byte[] is held in memory and not suitable for large potions of data.

Writing to an new ObjectOutputStream(new FileOutputStream(filename)), you can get your objects back using an new ObjectInputStream(new FileInputStream(filename)). If you want, you can also do some buffering as mentioned in Christopher's answer.

score 0 · Answer 3 · edited May 23 '17 at 10:25

0

You could probably use the method here: byte array to short array and back again in java and just write and read the bytearray. I would also recommend using BufferedOutputStream instead of just a FileOutputStream.

edited May 23 '17 at 10:25

Community

1
1

answered Aug 07 '13 at 17:18

Christopher

8,815
2
32
41

Saving large short array android

3 Answers3