C# and Kotlin ECDH shared key mismatch

Question

Deriving shared key using:

• C# ECDiffieHellmanCng.DeriveKeyMaterial(ECDiffieHellmanPublicKey otherPartyPublicKey)
• Kotlin KeyAgreement.generateSecret() followed by KeyAgreement.doPhase(key: Key!, lastPhase: Boolean)

Yields different results using curve "secp384r1".

Kotlin related links point to Kotlin for Android docs due to readability.

Simplified driver code to demonstrate the problem, assuming that C# .NET 7.0.1 console application is "Server" and Kotlin OpenJDK 19.0.1 application is "Client":

C#:

using System.Net;
using System.Net.Sockets;
using System.Security.Cryptography;

var listener = new TcpListener(IPAddress.Any, 13000);
listener.Start();
using var client = await listener.AcceptTcpClientAsync();
var sharedKey = await GetSharedKey(client, CancellationToken.None);

async Task<byte[]> GetSharedKey(TcpClient client, CancellationToken token)
{
    //Generate ECDH key pair using secp384r1 curve
    var ecdh = new ECDiffieHellmanCng(ECCurve.CreateFromFriendlyName("secp384r1"));
    var publicKeyBytes = ecdh.ExportSubjectPublicKeyInfo();
    Console.WriteLine($"Server Public Key: {Convert.ToBase64String(publicKeyBytes)}, " +
                      $"Length: {publicKeyBytes.Length}");

    //Send the generated public key encoded in X.509 to client.
    var stream = client.GetStream();
    await stream.WriteAsync(publicKeyBytes, token);
        
    //Receive client's public key bytes (X.509 encoding).
    var otherPublicKeyBytes = new byte[publicKeyBytes.Length];
    await stream.ReadExactlyAsync(otherPublicKeyBytes, 0, otherPublicKeyBytes.Length, token);
        
    //Decode client's public key bytes.
    var otherEcdh = new ECDiffieHellmanCng(ECCurve.CreateFromFriendlyName("secp384r1"));
    otherEcdh.ImportSubjectPublicKeyInfo(otherPublicKeyBytes, out _);
    Console.WriteLine($"Client Public Key: {Convert.ToBase64String(otherEcdh.ExportSubjectPublicKeyInfo())}, " +
                      $"Length: {otherEcdh.ExportSubjectPublicKeyInfo().Length}");

    //Derive shared key.
    var sharedKey = ecdh.DeriveKeyMaterial(otherEcdh.PublicKey);
    Console.WriteLine($"Shared key: {Convert.ToBase64String(sharedKey)}, " +
                      $"Length: {sharedKey.Length}");
    return sharedKey;
}

Kotlin:

import java.net.Socket
import java.security.KeyFactory
import java.security.KeyPairGenerator
import java.security.spec.ECGenParameterSpec
import java.security.spec.X509EncodedKeySpec
import java.util.*
import javax.crypto.KeyAgreement

fun main(args: Array<String>) {
    val socket = Socket("127.0.0.1", 13000)
    val sharedKey = getSharedKey(socket)
}

private fun getSharedKey(socket: Socket): ByteArray {
    //Generate ECDH key pair using secp384r1 curve
    val keyGen = KeyPairGenerator.getInstance("EC")
    keyGen.initialize(ECGenParameterSpec("secp384r1"))
    val keyPair = keyGen.generateKeyPair()
    println("Client Public Key: ${Base64.getEncoder().encodeToString(keyPair.public.encoded)}, Length: ${keyPair.public.encoded.size}")

    //Receive server's public key bytes (encoded in X.509)
    val input = socket.getInputStream()
    val publicKeyBytes = input.readNBytes(keyPair.public.encoded.size)

    //Send the generated public key encoded in X.509 to server
    val output = socket.getOutputStream()
    output.write(keyPair.public.encoded)

    // Decode the server's public key
    val keySpec = X509EncodedKeySpec(publicKeyBytes)
    val keyFactory = KeyFactory.getInstance("EC")
    val otherPublicKey = keyFactory.generatePublic(keySpec)
    println("Server Public Key: ${Base64.getEncoder().encodeToString(otherPublicKey.encoded)}, Length: ${otherPublicKey.encoded.size}")

    // Use KeyAgreement to generate the shared key
    val keyAgreement = KeyAgreement.getInstance("ECDH")
    keyAgreement.init(keyPair.private)
    keyAgreement.doPhase(otherPublicKey, true)
    val sharedKey = keyAgreement.generateSecret()
    println("Shared key: ${Base64.getEncoder().encodeToString(sharedKey)}, Length: ${sharedKey.size}")
    return sharedKey
}

C# output:

Server Public Key: MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEqza/eiK23hQIEW5mVdqOc0hAP3tPqittlcvPa6bGdyJK9n64sg0qYyDoPsxJ4pf7ROLz0ACrDS7n/e5Z0J1SMsWpBDViS8NRBvKwa1rQjWdFR0wzRaeVg09LIjnGs4Mj, Length: 120
Client Public Key: MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE30zvqkljT4STiE6XfLtoN147WRGA92rz9BLZfbRkOjz7uNbQ3az46DdoyQi6+eON7QVjIf2H5LKBANSk+C5zRX6u8jjrbhURDHYBKgijOddy6mOaEwiADijD/NX72O2L, Length: 120
Shared key: /u+tZYHar4MxXfrn2oqPZAqhiB2pkSTRBZ12rUxdnII=, Length: 32

Kotlin output:

Client Public Key: MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE30zvqkljT4STiE6XfLtoN147WRGA92rz9BLZfbRkOjz7uNbQ3az46DdoyQi6+eON7QVjIf2H5LKBANSk+C5zRX6u8jjrbhURDHYBKgijOddy6mOaEwiADijD/NX72O2L, Length: 120
Server Public Key: MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEqza/eiK23hQIEW5mVdqOc0hAP3tPqittlcvPa6bGdyJK9n64sg0qYyDoPsxJ4pf7ROLz0ACrDS7n/e5Z0J1SMsWpBDViS8NRBvKwa1rQjWdFR0wzRaeVg09LIjnGs4Mj, Length: 120
Shared key: lErK9DJAutaJ4af7EYWvtEXicAwfSuadtQhlZxug26wGkgB/ce7hF6ihLL87Sqc3, Length: 48

It seems there are no problems with public key import/export, but C# side fails to even produce key of correct length (384 / 8 = 48).

Edit: Somebody noticed that curiously enough C# "shared key" is Kotlin's shared key's SHA256 hash instead of the actual key.

I strongly suspect it's because of default key derivation function mismatch, but am not completely sure.

I would like to know what am I doing wrong and how to fix the issue.

Edit#2 - Solution: As the accepted answer suggests - my suspicion is not entirely wrong. ECDiffieHellmanCng.DeriveKeyMaterial does a bit extra unnecessary work - namely returning derived key's SHA256 hash (by default) instead of the actual key and does not provide any means of returning the actual key.

For anyone that is interested in getting 48 byte shared key you will have to be content with it's SHA384 (or some other hashing algorithm) hash instead (or use BouncyCastle):

C# changes:

//Generate ECDH key pair using secp384r1 curve and change default key's hashing algorithm SHA256 to SHA384
var ecdh = new ECDiffieHellmanCng(ECCurve.CreateFromFriendlyName("secp384r1"))
{
    HashAlgorithm = CngAlgorithm.Sha384
};

Kotlin changes:

val sharedKey = keyAgreement.generateSecret()
val sharedKeyHash = MessageDigest.getInstance("SHA384").digest(sharedKey)
println("Shared key SHA384 hash: ${Base64.getEncoder().encodeToString(sharedKeyHash)}, Length: ${sharedKeyHash.size}")
return sharedKeyHash

I also suggest to rename the GetSharedKey method to what it actually is - GetSharedKeysSHA384Hash.

Answer 1

The problem is that C# does do more than what is expected from the class. I.e. as usual, the .NET library doesn't adhere to the principle of least surprise:

The ECDiffieHellmanCng class enables two parties to exchange private key material even if they are communicating through a public channel. Both parties can calculate the same secret value, which is referred to as the secret agreement in the managed Diffie-Hellman classes. The secret agreement can then be used for a variety of purposes, including as a symmetric key. However, instead of exposing the secret agreement directly, the ECDiffieHellmanCng class does some post-processing on the agreement before providing the value. This post processing is referred to as the key derivation function (KDF); you can select which KDF you want to use and set its parameters through a set of properties on the instance of the Diffie-Hellman object.

Of course, the tw... developers that created the code don't exactly specify on what they perform the KDF, nor do they specify the default method used from the options that are shown. However, you can expect that they perform it on the X coordinate that is calculated by the Diffie-Hellman key agreement.

That said, it is not very clear from the Java class description either. The standard names document references RFC 3278, which points to the old Sec1 standard, section 6.1 using a broken link. Now Sec1 can still be downloaded, and if we look at section 6.1 we find a construction to encode integers to the a number of bytes that is the field size (and then take the required bytes). What however is returned is undoubtedly the same encoded X-coordinate that is the Input Keying Material to the KDF that Microsoft uses.

Phew, that was a lot of words to say that you have to take the result of the Kotlin code in bytes and then perform the SHA-256 algorithm on it. Oh, yeah, the SHA-256 default was guessed, it's also not specified as far as I can see by Microsoft, although they do expose the KeyDerivationFunction and HashAlgorithm properties and define the defaults for them.

There are some options to choose the parameters for the various KDF functions for ECDiffieHellmanCng, but you seem to be out of luck if you want to have the "raw" X-coordinate. If you want that you may need to use Bouncy Castle for C# but beware that it returns a raw integer for the X-coordinate instead of an encoding of a statically sized, unsigned, big endian integer.

Answer 2

We've stumpled upon this topic recently, when a certain interface out of our control required us to use a key based upon the unhashed raw secret with a given length (32 bytes e.g.), requiring padding or trimming of some 0x00s, which isnt possible without the raw secret itself.

AFAIK they are trying to implement it in .NET 8, so if your free to choose your environment, you may drop BC at a given time... https://github.com/dotnet/runtime/issues/71613