Successful handshake despite bad certificate in Netty 4.1.36 and jdk 11

Question

I have client and server configured to use TLS and self-signed certificate.

Client SSL Engine is configured to use dummy trust manager, which doesn't ever throw CertificateException and empty KeyManager array.

Server SSL Engine uses key store which is initialized with manually generated key store file.

When I run it with JDK 8 I get the following handshake result:

• Server fails to validate certificate
• In client's thread I can see that io.netty.handler.ssl.SslHandler#setHandshakeFailure is called and io.netty.handler.ssl.SslHandler#setHandshakeSuccess is never called.

Which is expected behavior.

When I run it with JDK 11 I get the following:

• server fails with the same error (empty certificate chain) but in client thread I see the following:
• io.netty.handler.ssl.SslHandler#setHandshakeSuccess is called first
• io.netty.handler.ssl.SslHandler#setHandshakeFailure is called after

I'm new to TLS 1.3 and might missed something in configuration. At the same time documentation says there's nothing to update for java TLS API clients to switch to TLS 1.3.

This behavior is confusing and it breaks further logic based on handshakePromise.

Full code to reproduce the issue is available by gist link: https://gist.github.com/kiturutin/ccb710f67ccfb0a7a7de1fb3b3099b60

It's a groovy script and it first starts server then client.

Answer 1

To be clear, your server requests client auth and server validation of the client cert fails because the client is configured with no keymanager and thus sends no cert? (Whereas server sent its own 'manual' cert okay, and client accepted it because of the dummy trustmanager.)

If so this looks like a similar problem jetty has(?) -- see the fifth post in https://github.com/eclipse/jetty.project/issues/2711 -- and it appears to me this is because TLS 1.3 client considers the handshake complete on sending Finished (because 1.3 moves server Finished to first flight) and that's before receiving the server alert for client auth failure. (Whereas in 1.2 and earlier, server Finished and possibly Ticket basically is the second flight, causing an extra RTT for client-first app protocols -- like HTTP.)

It is also before receiving server NewSessionTicket (as modified for 1.3 'resumption') if used; see OpenJDK 11 problem - Client finished handshake before last UNWRAP .

I guess for 1.3 you have to accept 'failure' after 'success', unless the Java folks can think of some really clever fix for this.