Convert byte array to BigInteger incorrect

Question

Since I tried to simulate the connection between my client and my server working on packet, I have been in the struggle with byte array and BigInteger.

When my client connected to the server, the server sent a response which include an RSA public key, then a packet in hex just like this:

C70001A100408031313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131E5AA194C3DBA71F9A38AF85D43C130A462BDCBACF85ED702B5E2FECB47197DAA433E7B5ACF5EF09A07C5DDE79B7A2EED76B91090CB981E13DBE316997FCF5DC51BE40B2D02D912457D97924B94094227F8BC65DD61FE78060B55BEAAA31C64E3B36863B407F7183DABA9D98F0C9061DED36AD16407F70C4925951BAA8807EC95

The 64 byte after C70001A1004080 is the RSA public exponent. The 128 byte next after RSA exponent is the modulus.

Then I tried to extract the exponent and the modulus to re-generate an RSA public key. My steps:

if (socket == null) socket = new Socket(loginHost, loginPort);
if (is == null) is = socket.getInputStream();
if (os == null) os = socket.getOutputStream();

int packageLength = 199;

byte[] response = IOUtils.toByteArray(is, packageLength); 

byte[] rsaPubExponentBytes = Arrays.copyOfRange(response, 7, 71);
byte[] rsaModuloBytes = Arrays.copyOfRange(response, 71, packageLength);

BigInteger rsaPubExponentNumber = new BigInteger(1, rsaPubExponentBytes);
BigInteger rsaModuloNumber = new BigInteger(1, rsaModuloBytes);

PublicKey pub = null;

try {
    RSAPublicKeySpec spec = new RSAPublicKeySpec(rsaModuloNumber, rsaPubExponentNumber);
    KeyFactory factory = KeyFactory.getInstance("RSA");
            
    pub = factory.generatePublic(spec);
} catch (NoSuchAlgorithmException | InvalidKeySpecException e) {
    e.printStackTrace();
}

Finally I had a public key:

Sun RSA public key, 1024 bits
  modulus: 161275860318613570660794349609651113455378591213521007305334206887210873414776462027746458127536797075994278751416175612148660222076731030937875746805281690156725876857482174279314968596913890067397869829419019179316569148699704292044128937649866682741942026327643537075317389490824101005022833230463069842581
  public exponent: 2576402308106616697565204803576809648025446765525597158856684355852417401857269811228595453373248109634555141377011045066015451991315223244608818828620081

Compare to the RSA from the server

Sun RSA public key, 1024 bits
  modulus: 105278801605955351183032861925660079240049371193037037019863031780531990060755731057393990664319512505978954889835933668032824049054190807125138251307247771506918060803959346902566451812689470408398611041302277800747341495594297701677567604614145074223483256879948522462707345079591319551080307454838134385381
  public exponent: 2576402308106616697565204803576809648025446765525597158856684355852417401857269811228595453373248109634555141377011045066015451991315223244608818828620081

Then the the modulus was wrong. I double checked by converting BigInteger to byte array and show it with hex

System.out.println(bytesToHex(rsaPubExponentNumber.toByteArray()));
System.out.println(bytesToHex(rsaModuloNumber.toByteArray()));

private String bytesToHex(byte[] bytes) {
    StringBuilder sb = new StringBuilder();
    for (byte b : bytes) {
       sb.append(String.format("%02X ", b));
    }
    return sb.toString().replaceAll(" ", "");
}

The result is:

Exponent: 31313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131

Modulus 00E5AA194C3DBA71F9A38AF85D43C130A462BDCBACF85ED702B5E2FECB47197DAA433E7B5ACF5EF09A07C5DDE79B7A2EED76B91090CB981E13DBE316997FCF5DC51BE40B2D02D912457D97924B94094227F8BC65DD61FE78060B55BEAAA31C64E3B36863B407F7183DABA9D98F0C9061DED36AD16407F70C4925951BAA8807EC95

The modulus now has 00 at the beginning and the public key was generated, but incorrect one.

I also tried:

BigInteger exponent = new BigInteger(exponentHexString, 16);
BigInteger modulus = new BigInteger(modulusHexString, 16);

However, the result still the same. How can I fix this?

Answer 1

Lets go through your code and packet stream structure.

You have the following binary stream encoded as a hex string:

C70001A100408031313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131E5AA194C3DBA71F9A38AF85D43C130A462BDCBACF85ED702B5E2FECB47197DAA433E7B5ACF5EF09A07C5DDE79B7A2EED76B91090CB981E13DBE316997FCF5DC51BE40B2D02D912457D97924B94094227F8BC65DD61FE78060B55BEAAA31C64E3B36863B407F7183DABA9D98F0C9061DED36AD16407F70C4925951BAA8807EC95

According to your code:

• bytes from index 7 to 71 are your exponent bytes,
• bytes from index 71 till the end of the stream are your modulus bytes.

IIUC, you want to extract the exponent and the modulus from the above byte string.

---

The issue you are facing stems from the fact that in cryptography, the numbers are typically unsigned integers, with a protocol-defined byte order, whereas Java's BigIntegers are big-endian, signed integers.

In order to convert an arbitrary integer (signed/unsigned, big-endian/little-endian) into a BigInteger, one has to perform some machinations.

In your case, it's enough to simply reverse the byte arrays, as suggested in the comments:

int packageLength = 199;

byte[] response = IOUtils.toByteArray(is, packageLength); 

byte[] rsaPubExponentBytes = Arrays.copyOfRange(response, 7, 71);
byte[] rsaModuloBytes = Arrays.copyOfRange(response, 71, packageLength);

// missing step!
reverse(rsaPubExponentBytes);
reverse(rsaModuloBytes);

BigInteger rsaPubExponentNumber = new BigInteger(1, rsaPubExponentBytes);
BigInteger rsaModuloNumber = new BigInteger(1, rsaModuloBytes);
// ...

public static void reverse(byte[] arr) {
    for (int i = 0; i < arr.length / 2) {
        int tmp = arr[i]
        arr[i] = arr[arr.length - 1 - i];
        arr[arr.length - 1 - i] = tmp;
    }
}

---

I've ran into a similar issue when working on a project of mine. You might find my library, SRP-6 Variables, helpful:

byte[] stream = new Hex("C70001A100408031313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131E5AA194C3DBA71F9A38AF85D43C130A462BDCBACF85ED702B5E2FECB47197DAA433E7B5ACF5EF09A07C5DDE79B7A2EED76B91090CB981E13DBE316997FCF5DC51BE40B2D02D912457D97924B94094227F8BC65DD61FE78060B55BEAAA31C64E3B36863B407F7183DABA9D98F0C9061DED36AD16407F70C4925951BAA8807EC95").asArray();

BigInteger exponent = 
    new SRP6CustomIntegerVariable(
        Bytes.wrapped(Arrays.copyOfRange(stream, 7, 71)),
        ByteOrder.LITTLE_ENDIAN
    ).asNonNegativeBigInteger();
BigInteger modulus = 
    new SRP6CustomIntegerVariable(
        Bytes.wrapped(Arrays.copyOfRange(stream, 71, stream.length)),
        ByteOrder.LITTLE_ENDIAN
    ).asNonNegativeBigInteger();