Write and read multiple byte[] in file with Java

Question

I have to write byte arrays in a file. I can't do it in one time so I can't put my arrays in a container. Also the size of my arrays is variable. Secondly, the file is very huge, so I have to split it, in order to read it array by array.

How can I do that ? I tried to write line by line my byte arrays but I haven't been able. How can I put a separator between my arrays and after split it over this separator ?

EDIT :

I tried this :

ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = new ObjectOutputStream(bos);
out.writeObject(byteArray);

But, I execute this code several times, so the ObjectOutputStream adds each time a new header which corrupt the file.

I also try :

out.write(byteArray);

but I couldn't separate my arrays. So I tried to append a '\n', which didn't work. After I was looking for library like FileUtils in order to write byte[] line by line but I didn't find.

Show some code, and tell us what you have tried. otherwise, I predict a downvote frenzy coming. — Colin D, Oct 19 '12 at 15:15
http://stackoverflow.com/questions/11110153/java-reading-file-chunk-by-chunk duplicate — smk, Oct 19 '12 at 15:24
@pith Have a look at second code snippet in the answer. I think this is what you need. — Amit Deshpande, Oct 19 '12 at 16:19

Amit Deshpande · Accepted Answer · 2012-10-19T16:21:12.887

You can use existing collections Like e.g. List to maintain List of byte[] and transfer it

    List<byte[]> list = new ArrayList<byte[]>();
    list.add("HI".getBytes());
    list.add("BYE".getBytes());

    ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(
            "test.txt"));
    out.writeObject(list);

    ObjectInputStream in = new ObjectInputStream(new FileInputStream(
            "test.txt"));
    List<byte[]> byteList = (List<byte[]>) in.readObject();

    //if you want to add to list you will need to add to byteList and write it again
    for (byte[] bytes : byteList) {
        System.out.println(new String(bytes));
    }

Output:

   HI
   BYE

Another option is use RandomAccessFile. Which will not force you to read complete file and you can skip the data that you don't want to read.

     DataOutputStream dataOutStream = new DataOutputStream(
            new FileOutputStream("test1"));
    int numberOfChunks = 2;
    dataOutStream.writeInt(numberOfChunks);// Write number of chunks first
    byte[] firstChunk = "HI".getBytes();
    dataOutStream.writeInt(firstChunk.length);//Write length of array a small custom protocol
    dataOutStream.write(firstChunk);//Write byte array

    byte[] secondChunk = "BYE".getBytes();
    dataOutStream.writeInt(secondChunk.length);//Write length of array
    dataOutStream.write(secondChunk);//Write byte array

    RandomAccessFile randomAccessFile = new RandomAccessFile("test1", "r");
    int chunksRead = randomAccessFile.readInt();
    for (int i = 0; i < chunksRead; i++) {
        int size = randomAccessFile.readInt();
        if (i == 1)// means we only want to read last chunk
        {
            byte[] bytes = new byte[size];
            randomAccessFile.read(bytes, 0, bytes.length);
            System.out.println(new String(bytes));
        }
        randomAccessFile.seek(4+(i+1)*size+4*(i+1));//From start so 4 int + i* size+ 4* i ie. size of i
    }

Output:

BYE

score 1 · Answer 2 · answered Oct 19 '12 at 15:19

1

You have to described your data in your encoding. i.e. add some metadata.

For example, the length of the array, then the data of the array.

This is called serialization.

Array of int: length(4 bytes), data[0] (4 bytes), data[1] (4 bytes), data[2] (4 bytes)

answered Oct 19 '12 at 15:19

Aubin

14,617
9
61
84

Really? That's *serialization* for you? – Sujay Oct 19 '12 at 15:20
It's a form of serialization, although you would normally only add as much meta-data as necessary to de-serialize and re-create the object (or array). – Peter Ilfrich Oct 19 '12 at 15:23

score 0 · Answer 3 · answered Oct 19 '12 at 16:03

There are several ways of doing this. Basically first thing to point out is that a file is just an unstructured sequence of bytes. Any bytes. Basically you want o store several byte arrays of variable length. That means you have to add structure in some way, which you then parse when you read.

The easiest is probably to use some sequence of bytes that you treat as a separator (specific enough that it has a low probability of showing up in your data otherwise).

Then on writing, first simply write your bytes, then the separator

out.write(myarray);
out.write(separator);
out.write(anotherarray);

On reading, you need to use some kind of sliding window that you can inspect so that you can figure out if you have read the separator. Basically just iterate over the bytes one by one and keep the last few you've read in some buffer. When you see the separator in your buffer, you have just found the end of your array.

An alternative to this could be writing blocks of a fixed length with some header that describes the length and maybe a bit for denoting whether there is another packet for the current array. Then you simply read and write whole blocks.

How to you deal with the problem: I have separator INTO an array? — Aubin, Oct 19 '12 at 17:08

Write and read multiple byte[] in file with Java

3 Answers3