how to decrypt radius peap protocol client finish handshake message

Question

i am using TLS_RSA_WITH_3DES_EDE_CBC_SHA cipher suite for the radius server, received a encrypted handshake message(40 bytes) right after ChangeCipherSpec from the client, i had tried to use 3des with cbc mode to decrypt those bytes, but with an exception(bad data), tried to look up the peap tls v1.0 on https://www.rfc-editor.org/rfc/rfc2246 but, didn't find a lot of infos about the finish handshake encryption/decryption in details. any help will be wonderful, thanks a lot!!

here are the code i used to compute the master secret and key materials.

    public static byte[] ComputeMasterSecret(byte[] pre_master_secret, byte[] client_random, byte[] server_random)
    {
        byte[] label = Encoding.ASCII.GetBytes("master secret");

        var seed = new List<byte>();
        seed.AddRange(client_random);
        seed.AddRange(server_random);

        var master_secret = PRF(pre_master_secret, label, seed.ToArray(), 48);

        return master_secret;
    }

    public static KeyMaterial ComputeKeys(byte[] master_secret, byte[] client_random, byte[] server_random)
    {
        /*
         * The cipher spec which is defined in this document which requires
        the most material is 3DES_EDE_CBC_SHA: it requires 2 x 24 byte
        keys, 2 x 20 byte MAC secrets, and 2 x 8 byte IVs, for a total of
 
        104 bytes of key material.
        */

 
        byte[] label = Encoding.ASCII.GetBytes("key expansion");

        var seed = new List<byte>();
        seed.AddRange(client_random);
        seed.AddRange(server_random);

        byte[] key_material = PRF(master_secret, label, seed.ToArray(), 104);  //need 104 for TLS_RSA_WITH_3DES_EDE_CBC_SHA cipher suite


        var km = new KeyMaterial();
        int idx = 0;
        km.ClientWriteMACSecret = Utils.CopyArray(key_material, idx, 20);
        idx += 20;

        km.ServerWriteMACSecret = Utils.CopyArray(key_material, idx, 20);
        idx += 20;

        km.ClientWriteKey = Utils.CopyArray(key_material, idx, 24);
        idx += 24;

        km.ServerWriteKey = Utils.CopyArray(key_material, idx, 24);
        idx += 24;

        km.ClientWriteIV = Utils.CopyArray(key_material, idx, 8);
        idx += 8;

        km.ServerWriteIV = Utils.CopyArray(key_material, idx, 8);

        return km;
    }

    public static byte[] PRF(byte[] secret, byte[] label, byte[] seed, int outputLength)
    {
        List<byte> s1 = new List<byte>();
        List<byte> s2 = new List<byte>();

        int size = (int)Math.Ceiling((double)secret.Length / 2);
        
        for(int i=0;i < size; i++)
        {
            s1.Add(secret[i]);
            s2.Insert(0, secret[secret.Length - i - 1]);
        }

        var tbc = new List<byte>();
        tbc.AddRange(label);
        tbc.AddRange(seed);

        var md5Result = MD5Hash(s1.ToArray(), tbc.ToArray(), outputLength);

        var sha1Result = SHA1Hash(s2.ToArray(), tbc.ToArray(), outputLength);
        

        var result = new List<byte>();
        for (int i = 0; i < outputLength; i++)
            result.Add((byte)(md5Result[i] ^ sha1Result[i]));

        return result.ToArray();
    }

    /// <summary>
    /// P_hash(secret, seed) = HMAC_hash(secret, A(1) + seed) +
    ///                        HMAC_hash(secret, A(2) + seed) +
    ///                        HMAC_hash(secret, A(3) + seed) + ...

    /// Where + indicates concatenation.

    ///  A() is defined as:
    /// A(0) = seed
    /// A(i) = HMAC_hash(secret, A(i-1))
    /// </summary>
    /// <param name="secret"></param>
    /// <param name="seed"></param>
    /// <param name="iterations"></param>
    /// <returns></returns>
    private static byte[] MD5Hash(byte[] secret, byte[] seed, int outputLength)
    {
        int iterations = (int)Math.Ceiling((double)outputLength / 16);

        HMACMD5 HMD5 = new HMACMD5(secret);

        var result = new List<byte>();
        byte[] A = null;
        for (int i = 0; i <= iterations; i++)
            if (A == null)
                A = seed;
            else
            {
                A = HMD5.ComputeHash(A);

                var tBuff = new List<byte>();
                tBuff.AddRange(A);
                tBuff.AddRange(seed);

                var tb = HMD5.ComputeHash(tBuff.ToArray());

                result.AddRange(tb);
            }

        return result.ToArray();
    }

    private static byte[] SHA1Hash(byte[] secret, byte[] seed, int outputLength)
    {
        int iterations = (int)Math.Ceiling((double)outputLength / 20);

        HMACSHA1 HSHA1 = new HMACSHA1(secret);
        var result = new List<byte>();
        byte[] A = null;

        for (int i = 0; i <= iterations; i++)
            if (A == null)
                A = seed;
            else
            {
                A = HSHA1.ComputeHash(A);

                var tBuff = new List<byte>();
                tBuff.AddRange(A);
                tBuff.AddRange(seed);

                var tb = HSHA1.ComputeHash(tBuff.ToArray());

                result.AddRange(tb);
            }

        return result.ToArray();
    }

Answer 1

The authentication/encryption and decryption/verification of the record containing the Finished handshake message are the same as all other records in SSL/TLS except that it is the first after CCS.

First (during handshake) the premaster secret from the keyexchange is used to derive a master secret and multiple working keys and IVs, depending on the suite. This varies a bit with the protocol version; for TLS1.0 see rfc2246 sections 8.1.1 (for plain RSA) 6.3 (for all keyexchanges) and 5.

Using a 'GenericBlock' (CBC) cipher -- which is the only option besides RC4 in TLS1.0 and 1.1 -- uses 6.2.3.1 fragmentation (not needed for this record) 6.2.3.2 optional compression (usually not used nowadays because of attacks like CRIME, and pretty useless for EAP traffic anyway) and 6.2.3.2.

Specifically, first an HMAC is added (for this suite HMAC-SHA1), then padding, then the result is encrypted using the data cipher (3DES-CBC) with an IV which for the first record (which Finished is) comes from the key derivation step above and for subsequent records comes from the last block of the previous record. (The latter is the flaw first reported by Rogaway and exploited by BEAST.)

Decryption and verification reverses this process in the obvious way. Note rfc2246 doesn't specify receiver must check all bytes of padding but this is apparently intended; rfc4346 (1.1) does specify it, without changing the content. (rfc4346 does change the IV handling to fix the Rogaway flaw. SSLv3 specified random padding except for the final length byte, so it can't be checked; this is the POODLE flaw.)

Some libraries/APIs that provide CBC mode for block ciphers (including 3DES) for arbitrary data default to PKCS5/7 padding. The padding TLS uses is similar to but NOT compatible with PKCS5/7 padding so using those libraries you may have to handle padding and unpadding yourself following the instructions in 6.2.3.2 -- or the code in any of about a dozen opensource TLS implementations.