It depends...
The SSL/TLS standard itself doesn't specify how and when the server certificate is verified.
From the introduction:
[...] the decisions on how to initiate TLS handshaking and how to interpret the authentication certificates exchanged are left to the judgment of the designers and implementors of protocols that run on top of TLS.
This being said, while it doesn't specify how the authentication has to take place, implementations are meant to perform this check during the handshake (or at the very least, immediately after):
- See Appendix D.
- Some error messages are clearly related to certificate authentication failure (
bad_certificate
, certificate_expired
, ...).
- Some of the text in the handshake overview: "[...] If the server is authenticated, it may request a certificate from the client, if that is appropriate to the cipher suite selected."
In most cases, the certificate verification itself is guided by RFC 3280/RFC 5280.
A number of SSL/TLS stacks will at least do this by default.
The host name verification, which could be considered as one of the certificate authentication step, has historically been implemented separately. This is mainly because RFC 3280/RFC 5280 didn't address this step and left it to each application protocol. There is a relatively recent harmonisation attempt in RFC 6125 (you can find the differences in protocols in Appendix B).
Whether host name checking is done during the SSL/TLS handshake depends on the library you're using, and how you've configured it.
For example, before Java 7, this had to be done separately from the main JSSE API (SSLSocket
/SSLEngine
). (This was done in HttpsURLConnection
, for example, but this sits on top of the JSSE, not within.) Since Java 7, it's possible to perform this check during the handshake and within the JSSE using the X509ExtendedTrustManager
, but this has to be configured using SSLParameters.setEndpointIdentificationAlgorithm(...)
, which only supports HTTPS
and LDAPS
(this being said, even if your service doesn't use HTTP, using HTTPS
for the endpoint identification algorithm wouldn't be a bad choice, certainly better than nothing).
Other SSL/TLS libraries or wrapping other libraries in other languages tend at least to have callbacks for this. Whether they're used (and used correctly) by developers depends, as shown in this paper. (You might also be interested in this question on Security.SE.)