What is the most efficient way to read a variable data length from binary input stream

Question

I am using a BufferedInputStream and reading some data fields from a binary file. I have a header that contains a field I must read to determine the number of bytes I need read for the next message. So the amount of data I need to read is always variable. I tried reading a the entire number of bytes for the message, at one time but this sometimes fails reading the entire size of the message.

So right now I am just looping and reading 1 byte at a time until I have the whole message of data. This seems to work but does seem to be the most efficient way of reading a big file of data(50 meg file).

What is the best way to read variable amount of data from streamed binary file?

Thanks!!

              int numRead= 0;
              int total =0;
              int readSize =56;  // actually read from header
              while(numRead>=0  && total < readSize){
                  // reading one byte at a time
                  //is there any way to read bigger chunks reliably
                  // since file is large
                  numRead= this.in.read(inBytes,total ,1); 
                  if(numRead>0){
                      total += numRead;
                  }
              }

BufferedInputStream buffStream = new BufferedInputStream( new FileInputStream(... — user3470688, Jun 23 '15 at 14:27
I'm not sure what you are asking. Are you asking how to read the whole file or how to check for this header and then change how much data you read after that? — MadConan, Jun 23 '15 at 14:28
this is BufferedInputStream buffStream = new BufferedInputStream( new FileInputStream(.. — user3470688, Jun 23 '15 at 14:32
what I am asking is the readSize I actually get from reading the file header. I just hard coded it to 56 in the example. What I am asking is, I am reading one byte at a time in my read, instead of reading bigger chunks of data. This seems to work good. Is there any way to read bigger chunks of data reliably because the file is large. — user3470688, Jun 23 '15 at 14:35

score 0 · Answer 1 · answered Jun 23 '15 at 14:33

0

I think the best solution is to combine your input streams with DataInputStream and then use readByte() method.

DataInputStream dis = new DataInputStream(new BufferedInputStream( new FileInputStream(...

answered Jun 23 '15 at 14:33

Can you explain why you think, that this is the best solution? – Tom Jun 23 '15 at 14:37
because, from what I understood, he needs to read a primitive type ... *A data input stream lets an application read primitive Java data type* – Jun 23 '15 at 14:38
1

Don't write that to me, add that to your answer. Since you say it might be the best solution you should explain it there. This makes your answer more complete. – Tom Jun 23 '15 at 17:32

score 0 · Accepted Answer · answered Jun 23 '15 at 14:40

You should be able to use the read(byte[], int, int) method to fill a byte array.

int numRead= 0;
int total =0;
int readSize =56;
while(numRead >= 0 && total < readSize) {
    numRead = this.in.read(inBytes, total, readSize - total); 
    if(numRead > 0) {
       total += numRead;
    }
}

This is similar to what you've got, though it doesn't limit to 1 byte at a time.

score 0 · Answer 3 · edited May 23 '17 at 12:22

An answer has been posted there: java-inputstream-size Java InputStream size

For example:

int numRead= 0;
int total =0;
int readSize = 1000; //you can put an bigger value here
while(numRead>0){
    numRead= this.in.read(inBytes,0,readSize); 
    total += numRead;
}

}

With this method, it is possible to read the inputStream by block of readSize bytes.

Note: This method only make it possible to read the size. The byte array will only contains the bytes obtained during the last iteration, as we overwrite it each time putting the offset to 0. Another point is that you must create an array of bytes with a size of (at least) readSize

Finally, there are some API which make it possible to do the process of getting all the bytes from an inputStream into a byte array like commons-io. See this answer

score 0 · Answer 4 · answered Jun 23 '15 at 14:42

If you simply want to read the file efficiently

    final int BUFFER_SIZE = 1 << 10;
    try {
        BufferedInputStream in = new BufferedInputStream(new FileInputStream(fileName),BUFFER_SIZE);
        int bytesRead = 0;
        int bytesToRead = 1;
        byte[] bytes = new byte[BUFFER_SIZE];
        while((bytesRead = in.read(bytes)) != -1){
            doSomethingWithContent(bytes,bytesRead);
        }
    } catch (IOException e) {
        e.printStackTrace();
    }

This reads up to the BUFFER_SIZE of bytes into the byte[] on each read. That will make the number of reads significantly smaller, which is why it's a lot faster.

What is the most efficient way to read a variable data length from binary input stream

4 Answers4