Well this is pretty complex question. I will try to explain some parts but avoiding as much detail as possible (even after that it will be pretty long).
How does authentication with certificate work?
If a holder of the private key signs some data, other participants can use the public key of the signer to validate the signature. This mechanism can be used for authentication. Private and public keys are stored in certificate where private key is kept safe on the holder machine whereas certificate with public key can be publicly available.
How does it relate to HTTPS?
WCF offers transport and message security. The difference between them is described here. The transport security in case of HTTP is HTTPS where only server needs issued certificate and client must to trust this certificate. This certificate is used both for authenticating server to the client and for establishing secure channel (which uses symmetric encryption).
HTTPS also offers variant called Mutual HTTPS where client must have also issued certificate and client uses the certificate to authenticate to the server.
How does message security work and what is a purpose of two certificates in that scenario?
In case of message security each message is signed, encrypted and authenticated separately = all these security informations are part of the message. In case of SOAP this is described by many specifications but generally you are interested in security bindings and X.509 Token profile.
Security binding is part of WS-SecurityPolicy assertions and it is describes how the message is secured. We have three bindings:
- Symmetric security binding - symmetric encryption
- Asymmetric security binding - asymmetric encryption
- Transport security binding - assertion that message must be send over HTTPS or other secured transport
X.509 Token profile specifies how to transport certificates (public keys) in messages and how to use them.
Now if you have symmetric security binding you need only server certificate because
- When client wants to send message to the server it will first generate random key.
- It will use this key to encrypt and sign request
- It will use service certificate to encrypt derived key and pass it to the request as well.
- When the server receives the message it will first use its private key to decrypt that key.
- It will use decrypted key to decrypt the rest of the message.
- It will also use the key to encrypt the response because client knows that key.
- Client will use the same key generated for request to decrypt the response
This is symmetric encryption which is much more faster then asymmetric encryption but key derivation should not be available in WS-Security 1.0. It is available in WS-Security 1.1. HTTPS internally works in similar way but the key is the same for the whole connection lifetime.
If you have asymmetric security binding you need two certificates:
- Initiator must have its own certificate used to sign requests and decrypt responses
- Recipient must have its own certificate used to decrypt requests and sign responses
That means following algorithm
- Initiator encrypts request with recipient's public key
- Initiator signs request with its private key
- Recipient uses initiator's public key to validate request signature
- Recipient uses its private key to decrypt request
- Recipient uses initiator's public key to encrypt response
- Recipient uses its private key to sign response
- Initiator uses recipient's public key to validate response signature
- Initiator uses its private key to decrypt response
The order of signing and encrypting can be changed - there is another WS-SecurityPolicy assertion which says what should be done first.
These were basics. It can be much more complex because message security actually allow you as many certificates as you want - you can for example use endorsing token to sign primary signature with another certificate etc.
