From the BigInteger docs
Constructor and Description
BigInteger(byte[] val)
Translates a byte array containing the two's-complement binary
representation of a BigInteger into a BigInteger.
The Two's-complement is the real reason.
Lets see how...
(Byte)0xef
in binary = 11101111
Now convert that back to Int and you get -17 (base 10) or -11 (base 16).
Now take a look at
byte[] ba = new byte[] {0, (byte)0xEF};
This has the (Byte)0xef
but prepended by 0. Which means this array has 00000000 11101111, which when converted gives the correct result.
Why was the previous case different?
Check out 2's complement rules - SO Answer, Mandatory Wikipedia link
Another way of thinking about this
0xEF in Decimal = 239
Range of Byte = -127 to 128
We have Overflow.
239 - 128 = 111
Now count this 111 from back (Numeric data types have this circular behaviour, again due to 2's complement representation).
For example: 129.toByte = -127
(129 - 128 = 1, count from back the 1st value = -127)
Shortcut to counting from back if x>128 && x<256 then x.toByte = (x - 128) - 128
Here x = 239 so x.toByte = -17