How do I read in hex values from a binary file and decipher some bytes containing bitflag values?

Question

I'm reading a bunch of bytes from a binary file. It is a RAR file. I'm interested in the 11th and 12th bytes of the file because the header specifications state:

HEAD_FLAGS Bit flags: 2 bytes

            0x0001  - Volume attribute (archive volume)
            0x0002  - Archive comment present
                      RAR 3.x uses the separate comment block
                      and does not set this flag.

            0x0004  - Archive lock attribute
            0x0008  - Solid attribute (solid archive)
            0x0010  - New volume naming scheme ('volname.partN.rar')
            0x0020  - Authenticity information present
                      RAR 3.x does not set this flag.

            0x0040  - Recovery record present
            0x0080  - Block headers are encrypted
            0x0100  - First volume (set only by RAR 3.0 and later)

            other bits in HEAD_FLAGS are reserved for
            internal use

The file that I'm playing with has 00 and 0D as positions 11 and 12 respectively.

I can't make sense of these two values as they are bit flags (which I have failed to understand).

I have these two values in byte array that is 12 bytes long. What I need to check in this sequence is whether the flag 0x0100 and 0x0001 is set or not.

I'm lost with this. Thanks.

I've inspected some of the files in a Hex editor and what i've seen is that 11th and 12th bytes need to be read together. That's why the specs list all the bit flags are 4 letter hex codes. Checking the bit flags individually yields incorrect results.

Assimilating as much information from the answers/tips, 've solved this in the following way:

FileInputStream fisFileInputStream = new FileInputStream((new File("C:\\testarchive.r00"));

byte[] bytHeader = new byte[20]; //The first 20 bytes are the RAR header.
fisFileInputStream.read(bytHeader);

short val=(short)( ((bytHeader[10]&0xFF)<<8) | (bytHeader[11]&0xFF) ); //Joining the two bytes into a short

System.out.println("Volume Attribute (0x0001): " + ((val & 0x0001) != 0));
System.out.println("First volume (0x0100): " + ((val & 0x0100) != 0));

I've tried this code with multiple RAR archives — spanned ones, non-spanned ones, the first file of a spanned archive, another file of a spanned archive.

The code itself works fine except for a very minor quirk. I get the opposite results from my hex values i.e.

When inspecting a file which is not the first file in a spanned archive, I get the volume atrribute 0x0001 as not set (false), and the "First volume" 0x100 as set (true).

When inspecting a file which is the first file in a spanned archive, I get the exact opposite results.

Now I modify my code to believe that the original specs are wrong (highly unlikely) and that 0x0001 means that is is the first file in a spanned archive and 0x0100 means that is a spanned archive, then it's all okay.

..but I guess I'm doing something wrong here with my bit flag logic. Any ideas?

This might help: http://stackoverflow.com/questions/1092411/java-checking-if-a-bit-is-0-or-1-in-a-long — biziclop, Jul 03 '12 at 18:13
That looked helpful. I thought I'd try this solution: http://stackoverflow.com/a/1092551/304151. Since my flags are in two bytes, should I sum them up when I'm initialising the `BigInteger` object. According to the JavaDocs it accepts a `Long`. Thanks- — Mridang Agarwalla, Jul 03 '12 at 18:26

score 3 · Accepted Answer · answered Jul 03 '12 at 18:37

The idea of bit flags is that you can cram a bunch of flags into a small number of bytes by adding them together.

In this case, they're not saying that each value you see in that list is a separate pair of bytes. They're all crammed into two bytes. If you were writing the file, you would add these values together to get the value to write.

For example, suppose we want to say "Archive lock attribute", "New volume naming scheme", and "Block headers encrypted". That means we want to set 0x0004, 0x0010, and 0x0080. So we add these all together, 0x0004 + 0x0010 + 0x0080 = 0x0094, and that's the value we write. For this to work, all values must turn out to be a single bit, or to put it another way, all must be powers of 2.

So this documentation doesn't describe 18 bytes. It only describes 2.

To read it, you have to perform AND operations (&) with the desired flag value. Life if you want to know if "Block headers encrypted" is set, read the value and AND with 0x0010. As this is a two-byte value, you either want to get it into a short or you have to pick the correct byte. Let's say for the sake of argument that we get it into a short. Then we'd say

if ((flags & 0x0010) != 0)

If that expression is true, then the bit is set. If it is false, the bit is not set.

BTW, if you read it as a stream of bytes, you can put them into a short by writing:

short flags = b[0] | b[1]<<8;

(You may need a cast in there. I forget.)

Or you may have to switch b[0] and b[1], depending on whether the file is written low-hi or hi-low.

Hi jay. I have some doubts that maybe you could help me clear. So they've mentioned 2 bytes i.e. 16bits. Does that mean, it could theoretically have 16 bit flags? Another thing was that, they've only listed out 9 flags and listed the rest as reserved. Does that mean, the first 8 bit flags are stored in the first byte and the last bit flag is in the second byte? One of the rar files I inspected had these binary values in the 11th and 12th bit: `00000001` (`0x01`) and `00000001` (`0x01`). How should I interpret these? Should I sum them up first by first? — Mridang Agarwalla, Jul 04 '12 at 07:38
Yes, 2 bytes could hold 16 bit flags, 8 in each byte. In this case they've defined 9 flags, 8 in one byte and 1 in the other. I don't know whether the low byte is stored first in this file or the high byte. You'd have to have some further documentation or look at some examples to figure that out. — Jay, Jul 05 '12 at 07:03
HI jay, I made a detailed edit yesterday to my question. I've solved the issue but the results are opposite for the flags `0x0100` and `0x0001`. Could this occur because of the endianess of the file/system? Thanks. — Mridang Agarwalla, Jul 05 '12 at 07:11

Michael Besteck · Answer 2 · 2015-12-30T19:22:39.013

2

      FileInputStream fis = new FileInputStream("file.rar");
      byte[] ba = new byte[13];
      fis.read(ba);
      byte b11 = ba[11];
      byte b12 = ba[12];
      boolean flagIs0x10 = b12 == 0x10;
      System.out.println("flag is 0x10 = "+flagIs0x10);

If both bytes together should have the value 0x10, i.e. if it is a 16bit-word, then

      boolean flagIs0x10 = b11 == 0 && b12 == 0x10;

or

      boolean flagIs0x10 = b12 == 0 && b11 == 0x10;

edited Dec 30 '15 at 19:22

answered Jul 03 '12 at 18:26

Michael Besteck

2,415
18
10

+1 Note though that the question said `11th and 12th` bytes, which would be ba[10] and ba[11] respectively. – biziclop Jul 03 '12 at 18:29
It was "positions 11 and 12" and i assumed you count from 0. If it was meant counting from one, then replace in the answer 13 by 12, 12 by 11 and 11 by 10. – Michael Besteck Jul 03 '12 at 18:31
Michael, are you missing a `==` in the last two snippets. You've used a `=`. – Mridang Agarwalla Jul 03 '12 at 19:48
@Mridang Agarwalla: Thanks for the hint, took a while since i found the time to review, but corrected it. – Michael Besteck Dec 30 '15 at 19:24

score 0 · Answer 3 · answered Jul 03 '12 at 20:08

Here's an easy way to deal with the arithmetic. The following utility function modifies a java BitSet to set bit flags based on the contents of a byte value.

static void setByte(BitSet bitSet, int byteNum, byte b) {
    int base = byteNum * 8;
    bitSet.set(base + 7, (b & 0x80) != 0);
    bitSet.set(base + 6, (b & 0x40) != 0);
    bitSet.set(base + 5, (b & 0x20) != 0);
    bitSet.set(base + 4, (b & 0x10) != 0);
    bitSet.set(base + 3, (b & 0x08) != 0);
    bitSet.set(base + 2, (b & 0x04) != 0);
    bitSet.set(base + 1, (b & 0x02) != 0);
    bitSet.set(base + 0, (b & 0x01) != 0);
}

It takes a "byteNum" parameter, in case you want to have multiple bytes put into the bitset, which in your case you do.

byte 0 would be bitset positions 0-7 byte 1 would be bitset positions 8-15 etc....

And within each byte, the high order bit is position 7, low order bit position 0.

Then you could simply do this once you have the two bytes. I'm assuming that the first byte is the high order bits, (0x0800 - 0x8000), and the second byte is the low order bits (0x0001 - 0x0080), though your spec should tell you that.

byte buf[] = new byte[2];
inputStream.read(buf, 0, 2);

BitSet bitSet = new BitSet();
// low order byte is the second one
setByte(bitSet, 0, bytes[1]);
// high order byte is first
setByte(bitSet, 1, byte1[0]);

boolean archiveVolume = bitSet.get(0);
boolean commentPresent = bitSet.get(1);
...
boolean firstVolume = bitSet.get(8);

How do I read in hex values from a binary file and decipher some bytes containing bitflag values?

3 Answers3

Linked