OpenJDK 11 problem - Client finished handshake before last UNWRAP

Question

I am migrating our code base from Oracle Java 1.8.0_131 to OpenJDK 11.0.1. We have code that implements nio-ssl socket channels. In Java 8, the client/server handshake works fine. In Java 11, the client finishes the handshake before it unwraps the last handshake message from the server.

To produce this problem, I am just establishing a connection between the client and server and letting them perform a SSL handshake. I am NOT sending any additional data through.

I establish the connection using java 8 and get the output below. I then compile, build and run the same code using java 11 and get the other output below. I do not change any of my code.

I have some logging on both the client and server to show which step in the handshake they are in.

Log output Java 8 - client

SSL Handshake Started
WRAP:OK -   BytesProduced=172  BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=2295
TASK
WRAP:OK -   BytesProduced=1815 BytesConsumed=0
WRAP:OK -   BytesProduced=269  BytesConsumed=0
WRAP:OK -   BytesProduced=6    BytesConsumed=0
WRAP:OK -   BytesProduced=85   BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=85
SSL Handshake complete

Log output Java 8 - server

SSL Handshake Started
UNWRAP:OK - BytesProduced=0    BytesConsumed=172
TASK
WRAP:OK -   BytesProduced=2295 BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=1815
TASK
UNWRAP:OK - BytesProduced=0    BytesConsumed=269
TASK
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=85
WRAP:OK -   BytesProduced=6    BytesConsumed=6
WRAP:OK -   BytesProduced=85   BytesConsumed=0
SSL Handshake complete

Log output Java 11 - client

SSL Handshake Started
WRAP:OK -   BytesProduced=422  BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=160
TASK
WRAP:OK -   BytesProduced=6    BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=2204
TASK
WRAP:OK -   BytesProduced=2067 BytesConsumed=0
SSL Handshake complete
UNWRAP:OK - BytesProduced=0    BytesConsumed=72

Log output Java 11 - server

SSL Handshake Started
UNWRAP:OK - BytesProduced=0    BytesConsumed=422
TASK
WRAP:OK -   BytesProduced=160  BytesConsumed=0
WRAP:OK -   BytesProduced=6    BytesConsumed=0
WRAP:OK -   BytesProduced=2204 BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=2067
TASK
WRAP:OK -   BytesProduced=72    BytesConsumed=0
SSL Handshake complete

Code for handshaking

engine.beginHandshake();
HandshakeStatus hs = engine.getHandshakeStatus();
while(hs != HandshakeStatus.FINISHED && hs != HandshakeStatus.NOT_HANDSHAKING){
  switch(hs){
    case NEED_WRAP:
      SSLEngineResult res = engine.wrap(myAppData, myNetData)
      hs = res.getHandshakeStatus();
      switch(res.getStatus()){
        case OK:
          // write myNetData
        case BUFFER_OVERFLOW:
          // increase size of myNetData
        case BUFFER_UNDERFLOW:
          // throw exception
        case CLOSED:
          // clean up
        default:
          // throw illegal state exception
      }
      break;
    case NEED_UNWRAP:
      boolean complete = false;
      while(!complete){
        /*
         * First handle any encrypted data left on buffer
         * If there is none, read in more
         */
        if(peerNetData.position() > 0 || channel.read(peerNetData) > 0){
          peerNetData.flip();
          res = engine.unwrap(peerNetData, peerAppData);
          hs = res.getHandshakeStatus();
          switch(res.getStatus()){
            case OK:
              complete = true;
              peerNetData.compact();
              break;
            case BUFFER_UNDERFLOW:
              // if buffer is full, increase size
              // if buffer isn't full, compact and read
            case BUFFER_OVERFLOW:
              // increase size of peerAppData
            case CLOSED:
              // cleanup
            default:
              // throw illegal state exception
          }
        }
      }
      break;
    case NEED_TASK:
      // Run task
      hs = engine.getHandshakeStatus();
      break;
    case FINISHED:
      break;
    case NOT_HANDSHAKING:
      break;
    default:
      // illegal state
  }
}

Unfortunately, my code resides in an air-gapped environment, so pasting it here is not easy. I typed it in by hand so parenthesis and tabs might not line up.

The main point is the hs = res.getHandshakeStatus(...) returns FINISHED on the client machine after the wrap of 2067 bytes when it seems like it should return NEED_UNWRAP. If I change it to hs = engine.getHandshakeStatus(), it returns NOT_HANDSHAKING.

On the server machine, hs = engine.getHandshakeStatus() returns NEED_WRAP after running that last task causing it to WRAP that last 72 bytes.

Why does the SSLEngine on my client machine give me a handshake status of "FINISHED" when there are still 72 bytes of data to UNWRAP from the server? Has anyone else had any issues with custom handshake logic for Java 11?

j11 (and no earlier version) supports TLS1.3, which is being used in your example and makes significant changes to the handshake, particularly using a modified NewTicket in nearly all cases rather than very few as before, and in my testing with SSLSocket (which is inherently synchronous) JSSE client apparently treats 1.3 NewTicket as being after the end of the handshake, possibly since it cannot alter the outcome or session object contents. — dave_thompson_085, Dec 21 '18 at 08:25
Thanks for pointing that out, I forgot that Java11 added support for TLSv1.3 (which is what it defaults to). I will have to read up more on NewTickets because I don't fully understand what you are talking about. Anyway, when configuring my SSLSession to use TLSv1.2, the handshake works just fine. This can be my workaround for now... — user3335078, Dec 21 '18 at 14:05

score 3 · Answer 1 · answered Dec 21 '18 at 00:08

3

The JavaDoc of SSLEngine has a note about concurrency at the end of the class description.

Thus I assume your issue looks like a concurrency issue based on a race condition. It can differ how wrap and unwrap functions are called simultaneously after you migrated from JDK 8 to JDK 11. The example code you supplied doesn't have any synchronized statements.

If you synchronize both function calls with a shared object the handshake should finish every time after the last unwrap.

answered Dec 21 '18 at 00:08

fireandfuel

732
1
9
22

I should have mentioned that my code to establish the connection is single-threaded, so there are no instances of concurrent calls to SSLEngine's wrap and unwrap methods. – user3335078 Dec 21 '18 at 14:04
@user3335078 please create a SSCCE to reproduce it. If you think this is a bug, feel free to leave a bug report at https://bugs.java.com/bugdatabase/ – fireandfuel Dec 22 '18 at 16:01

score 0 · Answer 2 · edited Oct 07 '21 at 11:10

For anyone looking for a solution of this problem with SSLEngine, the cause of this problem is post-handshake messages defined in TLS 1.3: https://www.rfc-editor.org/rfc/rfc8446#section-4.6 To properly handle such message in your app, you have to check for handshake status of unwrap calls, if status is FINISHED, but handshake has already been done, it most likely caused by post-handshake message and data must be read and unwrapped again. See also: https://stackoverflow.com/a/57409789/6561198

OpenJDK 11 problem - Client finished handshake before last UNWRAP

2 Answers2

Linked