1

We have a service that makes a REST call to the USPS address validation service. We recently upgraded to a new version of our e-commerce framework (Broadleaf Commerce) with associated updates to Spring Boot and Java version. Since the upgrade, we are no longer able to connect to the USPS service. We fail as the SSL Handshake does not complete (seems the change cipher request is never answered). The existing production version of the code works as-is.

The weird part is that it works locally and if I make a cURL request from the target server, it works. But when I make the call from the application on the target server, it does not complete the handshake.

I have been searching the internet and stack overflow for answers but have not found one that fixes my issues. Here are the troubleshooting steps I have followed to date:

  1. I have tested the code locally and the USPS call works
  2. I have tested the server firewall / networking by executing a cURL command from the server. It worked as expected (TLS was negotiated properly)
  3. I dockerized the application to validate that the java versions and Java Trust Store certificates were consistent across environments. The dockerized application works on my local machine, it fails on the target machines with the same SSL Handshake errors.
  4. I have tested another REST call to another SSL secured endpoint. It worked properly. The differences between it and the USPS call are (1) the second endpoint is internal to our network and (2) the internal endpoint uses TLS 1.2, USPS uses TLS 1.3. Both are supported by the application per the logs.
  5. I attempted to force the JVM supported TLS versions using the jdk.tls.client.protocols and https.protocols properties. This had no effect (which was expected as TLS 1.2 and 1.3 are the current defaults).
  6. I have done a WireShark trace on both the cURL command and the Application call. WireShark shows cURL negotiating the Handshake properly. For the app, it seems that the request to change ciphers is never answered.
  7. I have verified with USPS that they have not blocked our IPs or anything like that.

I am out of ideas of what to test to identify the root cause of the issue. Doesn't seem to be code-related (the code works locally), it doesn't seem to be server network/firewall related (cURL works on the server).

What is left? What am I missing?

Error Message:

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://stg-secure.shippingapis.com/ShippingAPI.dll": Remote host terminated the handshake; nested exception is javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake

RestCall Code:

@Value("${usps.addressValidation.url}")
private String url;

private ResponseEntity<String> callUsps() {
Map<String, String> uriVariables = new HashMap<>();
    uriVariables.put("userId", apiKey);
    ... more variables ...

    ResponseEntity<String> response = restTemplate.postForEntity(url, null, String.class, uriVariables);
}

RestTemplate config:

@Bean
public RestTemplate restTemplate() {
    RestTemplate restTemplate = new RestTemplate();
    restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory()); // HttpComponents added for debug logging purposes, originally it was stock rest template
    return restTemplate;
}

Log output:

2023-03-09 20:43:39.007 DEBUG 1 --- [io-8080-exec-11] o.s.web.client.RestTemplate              : HTTP POST https://stg-secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=%3CAddressValidateRequest%20USERID%3D%22<userId>%22%20PASSWORD%3D%22<password>%22%3E%3CRevision%3E1%3C/Revision%3E%3CAddress%20ID%3D%220%22%3E%3CAddress1%3E%3C/Address1%3E%3CAddress2%3E<Address>%3C/Address2%3E%3CCity%3E<City>%3C/City%3E%3CState%3E<State>%3C/State%3E%3CZip5%3E<Zip>%3C/Zip5%3E%3CZip4%3E%3C/Zip4%3E%3C/Address%3E%3C/AddressValidateRequest%3E
2023-03-09 20:43:39.007 DEBUG 1 --- [io-8080-exec-11] o.s.web.client.RestTemplate              : Accept=[text/plain, application/xml, text/xml, application/json, application/x-jackson-smile, application/*+xml, application/*+json, */*]
2023-03-09 20:43:39.008 DEBUG 1 --- [io-8080-exec-11] o.a.h.client.protocol.RequestAddCookies  : CookieSpec selected: default
2023-03-09 20:43:39.008 DEBUG 1 --- [io-8080-exec-11] o.a.h.client.protocol.RequestAuthCache   : Auth cache not set in the context
2023-03-09 20:43:39.008 DEBUG 1 --- [io-8080-exec-11] h.i.c.PoolingHttpClientConnectionManager : Connection request: [route: {s}->https://stg-secure.shippingapis.com:443][total available: 1; route allocated: 0 of 5; total allocated: 1 of 10]
2023-03-09 20:43:39.008 DEBUG 1 --- [io-8080-exec-11] h.i.c.PoolingHttpClientConnectionManager : Connection leased: [id: 4][route: {s}->https://stg-secure.shippingapis.com:443][total available: 1; route allocated: 1 of 5; total allocated: 2 of 10]
2023-03-09 20:43:39.008 DEBUG 1 --- [io-8080-exec-11] o.a.http.impl.execchain.MainClientExec   : Opening connection {s}->https://stg-secure.shippingapis.com:443
2023-03-09 20:43:39.047 DEBUG 1 --- [io-8080-exec-11] .i.c.DefaultHttpClientConnectionOperator : Connecting to stg-secure.shippingapis.com/72.21.81.70:443
2023-03-09 20:43:39.047 DEBUG 1 --- [io-8080-exec-11] o.a.h.c.ssl.SSLConnectionSocketFactory   : Connecting socket to stg-secure.shippingapis.com/72.21.81.70:443 with timeout 0
2023-03-09 20:43:39.070 DEBUG 1 --- [io-8080-exec-11] o.a.h.c.ssl.SSLConnectionSocketFactory   : Enabled protocols: [TLSv1.3, TLSv1.2]
2023-03-09 20:43:39.071 DEBUG 1 --- [io-8080-exec-11] o.a.h.c.ssl.SSLConnectionSocketFactory   : Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2023-03-09 20:43:39.071 DEBUG 1 --- [io-8080-exec-11] o.a.h.c.ssl.SSLConnectionSocketFactory   : Starting handshake
2023-03-09 20:44:19.722 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing expired connections
2023-03-09 20:44:19.722 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 50000 MILLISECONDS
2023-03-09 20:44:19.723 DEBUG 1 --- [nection evictor] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-1: Close connection
2023-03-09 20:44:19.723 DEBUG 1 --- [nection evictor] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-0: Close connection
2023-03-09 20:44:19.731 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing expired connections
2023-03-09 20:44:19.731 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 50000 MILLISECONDS
2023-03-09 20:44:25.397 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing expired connections
2023-03-09 20:44:25.397 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 50000 MILLISECONDS
2023-03-09 20:44:25.401 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing expired connections
2023-03-09 20:44:25.402 DEBUG 1 --- [nection evictor] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 50000 MILLISECONDS
2023-03-09 20:44:39.520 DEBUG 1 --- [io-8080-exec-11] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-4: Shutdown connection
2023-03-09 20:44:39.520 DEBUG 1 --- [io-8080-exec-11] o.a.http.impl.execchain.MainClientExec   : Connection discarded
2023-03-09 20:44:39.520 DEBUG 1 --- [io-8080-exec-11] h.i.c.PoolingHttpClientConnectionManager : Connection released: [id: 4][route: {s}->https://stg-secure.shippingapis.com:443][total available: 1; route allocated: 0 of 5; total allocated: 1 of 10]
2023-03-09 20:44:39.521 ERROR 1 --- [io-8080-exec-11] b.c.b.s.v.a.AddressValidationServiceImpl : Exception while validating address.
2023-03-09 20:44:39.527 ERROR 1 --- [io-8080-exec-11] b.c.b.s.v.a.AddressValidationServiceImpl : Root exception: 

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://stg-secure.shippingapis.com/ShippingAPI.dll": Remote host terminated the handshake; nested exception is javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:732)
    at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:477)
    at company.commerce.broadleaf.wsclient.usps.service.UspsAddressValidationServiceClientImpl.callUsps(UspsAddressValidationServiceClientImpl.java:76)
    at company.commerce.broadleaf.wsclient.usps.service.UspsAddressValidationServiceClientImpl.validateAddress(UspsAddressValidationServiceClientImpl.java:62)
    at company.commerce.broadleaf.service.validation.address.AddressValidationServiceImpl.standardizeAddress(AddressValidationServiceImpl.java:85)
    at company.commerce.broadleaf.controller.account.AddressVerificationController.validateShippingAddress(AddressVerificationController.java:40)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
...
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
    at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1697)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1515)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
    ... 201 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
    at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
    at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)
    ... 220 common frames omitted

WireShark Output (failed call from Java app):

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       <Our IP>              <USPS IP>             TCP      74     49116 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2640145959 TSecr=0 WS=128
      2 0.023028       <USPS IP>             <Our IP>              TCP      74     443 → 49116 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1380 SACK_PERM TSval=3604861692 TSecr=2640145959 WS=512
      3 0.023028       <Our IP>              <USPS IP>             TCP      66     49116 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2640145982 TSecr=3604861692
      4 0.023836       <Our IP>              <USPS IP>             TLSv1.3  481    Client Hello
      5 0.024180       <USPS IP>             <Our IP>              TCP      66     443 → 49116 [ACK] Seq=1 Ack=416 Win=32768 Len=0 TSval=3604861692 TSecr=2640145982
      6 0.047150       <USPS IP>             <Our IP>              TLSv1.3  159    Hello Retry Request
      7 0.047151       <USPS IP>             <Our IP>              TLSv1.3  72     Change Cipher Spec
      8 0.047151       <Our IP>              <USPS IP>             TCP      66     49116 → 443 [ACK] Seq=416 Ack=94 Win=64256 Len=0 TSval=2640146006 TSecr=3604861715
      9 0.047175       <Our IP>              <USPS IP>             TCP      66     49116 → 443 [ACK] Seq=416 Ack=100 Win=64256 Len=0 TSval=2640146006 TSecr=3604861715
     10 0.048275       <Our IP>              <USPS IP>             TLSv1.3  514    Client Hello
     11 0.048280       <USPS IP>             <Our IP>              TCP      66     443 → 49116 [ACK] Seq=100 Ack=864 Win=32256 Len=0 TSval=3604861716 TSecr=2640146007

WireShark Output (successful call from cURL):

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       <Our IP>              <USPS IP>             TCP      74     58184 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3286965270 TSecr=0 WS=128
      2 0.023011       <USPS IP>             <Our IP>              TCP      74     443 → 58184 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1380 SACK_PERM TSval=401053718 TSecr=3286965270 WS=512
      3 0.023011       <Our IP>              <USPS IP>             TCP      66     58184 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3286965293 TSecr=401053718
      4 0.029806       <Our IP>              <USPS IP>             TLSv1.3  583    Client Hello
      5 0.030079       <USPS IP>             <Our IP>              TCP      66     443 → 58184 [ACK] Seq=1 Ack=518 Win=32768 Len=0 TSval=401053718 TSecr=3286965300
      6 0.252624       <USPS IP>             <Our IP>              TLSv1.3  159    Hello Retry Request
      7 0.252625       <USPS IP>             <Our IP>              TLSv1.3  72     Change Cipher Spec
      8 0.252625       <Our IP>              <USPS IP>             TCP      66     58184 → 443 [ACK] Seq=518 Ack=94 Win=64256 Len=0 TSval=3286965523 TSecr=401053941
      9 0.252625       <Our IP>              <USPS IP>             TCP      66     58184 → 443 [ACK] Seq=518 Ack=100 Win=64256 Len=0 TSval=3286965523 TSecr=401053941
     10 0.252852       <Our IP>              <USPS IP>             TLSv1.3  589    Change Cipher Spec, Client Hello
     11 0.252853       <USPS IP>             <Our IP>              TCP      66     443 → 58184 [ACK] Seq=100 Ack=1041 Win=31744 Len=0 TSval=401053941 TSecr=3286965523
     12 0.277201       <USPS IP>             <Our IP>              TLSv1.3  1434   Server Hello, Application Data
     13 0.277203       <USPS IP>             <Our IP>              TCP      1434   443 → 58184 [PSH, ACK] Seq=1468 Ack=1041 Win=68096 Len=1368 TSval=401053972 TSecr=3286965523 [TCP segment of a reassembled PDU]
     14 0.277269       <USPS IP>             <Our IP>              TLSv1.3  1354   Application Data, Application Data, Application Data
     15 0.277271       <Our IP>              <USPS IP>             TCP      66     58184 → 443 [ACK] Seq=1041 Ack=4124 Win=62592 Len=0 TSval=3286965547 TSecr=401053972
     16 0.278039       <Our IP>              <USPS IP>             TLSv1.3  140    Application Data
     17 0.278385       <Our IP>              <USPS IP>             TLSv1.3  112    Application Data
     18 0.278385       <Our IP>              <USPS IP>             TLSv1.3  115    Application Data
     19 0.278386       <Our IP>              <USPS IP>             TLSv1.3  101    Application Data
     20 0.278386       <Our IP>              <USPS IP>             TLSv1.3  414    Application Data
     21 0.300775       <USPS IP>             <Our IP>              TCP      66     443 → 58184 [ACK] Seq=4124 Ack=1115 Win=68096 Len=0 TSval=401053996 TSecr=3286965548
     22 0.300776       <USPS IP>             <Our IP>              TLSv1.3  321    Application Data
     23 0.300776       <USPS IP>             <Our IP>              TLSv1.3  321    Application Data
     24 0.300777       <USPS IP>             <Our IP>              TLSv1.3  162    Application Data, Application Data
     25 0.300828       <Our IP>              <USPS IP>             TLSv1.3  97     Application Data
     26 0.301368       <USPS IP>             <Our IP>              TCP      66     443 → 58184 [ACK] Seq=4730 Ack=1593 Win=69120 Len=0 TSval=401053997 TSecr=3286965548
     27 0.301724       <USPS IP>             <Our IP>              TLSv1.3  97     Application Data
     28 0.343705       <Our IP>              <USPS IP>             TCP      66     58184 → 443 [ACK] Seq=1624 Ack=4761 Win=64128 Len=0 TSval=3286965614 TSecr=401053997
     29 0.364919       <USPS IP>             <Our IP>              TCP      66     443 → 58184 [ACK] Seq=4761 Ack=1624 Win=69120 Len=0 TSval=401054060 TSecr=3286965571
     30 0.365596       <USPS IP>             <Our IP>              TLSv1.3  849    Application Data, Application Data, Application Data
     31 0.365600       <Our IP>              <USPS IP>             TCP      66     58184 → 443 [ACK] Seq=1624 Ack=5544 Win=64128 Len=0 TSval=3286965636 TSecr=401054061
     32 0.365952       <Our IP>              <USPS IP>             TLSv1.3  90     Application Data
     33 0.366549       <Our IP>              <USPS IP>             TCP      66     58184 → 443 [FIN, ACK] Seq=1648 Ack=5544 Win=64128 Len=0 TSval=3286965637 TSecr=401054061
     34 0.388421       <USPS IP>             <Our IP>              TCP      66     443 → 58184 [ACK] Seq=5544 Ack=1648 Win=69120 Len=0 TSval=401054084 TSecr=3286965636
     35 0.388421       <USPS IP>             <Our IP>              TLSv1.3  90     Application Data
     36 0.388421       <USPS IP>             <Our IP>              TCP      66     443 → 58184 [FIN, ACK] Seq=5568 Ack=1648 Win=69120 Len=0 TSval=401054084 TSecr=3286965636
     37 0.388422       <Our IP>              <USPS IP>             TCP      60     58184 → 443 [RST] Seq=1648 Win=0 Len=0
     38 0.388422       <Our IP>              <USPS IP>             TCP      60     58184 → 443 [RST] Seq=1648 Win=0 Len=0
     39 80.823845      <Our IP>              <USPS IP>             TCP      74     58192 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3287046092 TSecr=0 WS=128
     40 80.846555      <USPS IP>             <Our IP>              TCP      74     443 → 58192 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1380 SACK_PERM TSval=472250826 TSecr=3287046092 WS=512
     41 80.846556      <Our IP>              <USPS IP>             TCP      66     58192 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3287046115 TSecr=472250826
     42 80.853620      <Our IP>              <USPS IP>             TLSv1.3  583    Client Hello
     43 80.854172      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [ACK] Seq=1 Ack=518 Win=32768 Len=0 TSval=472250826 TSecr=3287046122
     44 80.876813      <USPS IP>             <Our IP>              TLSv1.3  159    Hello Retry Request
     45 80.876818      <Our IP>              <USPS IP>             TCP      66     58192 → 443 [ACK] Seq=518 Ack=94 Win=64256 Len=0 TSval=3287046145 TSecr=472250848
     46 80.876818      <USPS IP>             <Our IP>              TLSv1.3  72     Change Cipher Spec
     47 80.876818      <Our IP>              <USPS IP>             TCP      66     58192 → 443 [ACK] Seq=518 Ack=100 Win=64256 Len=0 TSval=3287046145 TSecr=472250848
     48 80.877240      <Our IP>              <USPS IP>             TLSv1.3  589    Change Cipher Spec, Client Hello
     49 80.877240      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [ACK] Seq=100 Ack=1041 Win=31744 Len=0 TSval=472250849 TSecr=3287046146
     50 80.900887      <USPS IP>             <Our IP>              TLSv1.3  1434   Server Hello, Application Data
     51 80.900888      <USPS IP>             <Our IP>              TCP      1434   443 → 58192 [PSH, ACK] Seq=1468 Ack=1041 Win=68096 Len=1368 TSval=472250881 TSecr=3287046145 [TCP segment of a reassembled PDU]
     52 80.900888      <USPS IP>             <Our IP>              TLSv1.3  1354   Application Data, Application Data, Application Data
     53 80.901181      <Our IP>              <USPS IP>             TCP      66     58192 → 443 [ACK] Seq=1041 Ack=4124 Win=62592 Len=0 TSval=3287046170 TSecr=472250881
     54 80.902491      <Our IP>              <USPS IP>             TLSv1.3  140    Application Data
     55 80.903307      <Our IP>              <USPS IP>             TLSv1.3  112    Application Data
     56 80.903307      <Our IP>              <USPS IP>             TLSv1.3  115    Application Data
     57 80.903495      <Our IP>              <USPS IP>             TLSv1.3  101    Application Data
     58 80.903496      <Our IP>              <USPS IP>             TLSv1.3  414    Application Data
     59 80.924618      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [ACK] Seq=4124 Ack=1115 Win=68096 Len=0 TSval=472250905 TSecr=3287046171
     60 80.924630      <USPS IP>             <Our IP>              TLSv1.3  321    Application Data
     61 80.924725      <USPS IP>             <Our IP>              TLSv1.3  321    Application Data
     62 80.924725      <USPS IP>             <Our IP>              TLSv1.3  162    Application Data, Application Data
     63 80.925401      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [ACK] Seq=4730 Ack=1210 Win=68096 Len=0 TSval=472250906 TSecr=3287046172
     64 80.925401      <USPS IP>             <Our IP>              TLSv1.3  97     Application Data
     65 80.925649      <Our IP>              <USPS IP>             TLSv1.3  97     Application Data
     66 80.925817      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [ACK] Seq=4761 Ack=1593 Win=69120 Len=0 TSval=472250906 TSecr=3287046172
     67 80.987738      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [ACK] Seq=4761 Ack=1624 Win=69120 Len=0 TSval=472250968 TSecr=3287046194
     68 80.989877      <USPS IP>             <Our IP>              TLSv1.3  849    Application Data, Application Data, Application Data
     69 80.992126      <Our IP>              <USPS IP>             TLSv1.3  90     Application Data
     70 80.992648      <Our IP>              <USPS IP>             TCP      66     58192 → 443 [FIN, ACK] Seq=1648 Ack=5544 Win=64128 Len=0 TSval=3287046261 TSecr=472250970
     71 81.014173      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [ACK] Seq=5544 Ack=1648 Win=69120 Len=0 TSval=472250994 TSecr=3287046261
     72 81.014173      <USPS IP>             <Our IP>              TLSv1.3  90     Application Data
     73 81.014173      <USPS IP>             <Our IP>              TCP      66     443 → 58192 [FIN, ACK] Seq=5568 Ack=1648 Win=69120 Len=0 TSval=472250994 TSecr=3287046261
     74 81.014313      <Our IP>              <USPS IP>             TCP      60     58192 → 443 [RST] Seq=1648 Win=0 Len=0
     75 81.014314      <Our IP>              <USPS IP>             TCP      60     58192 → 443 [RST] Seq=1648 Win=0 Len=0
Marquee
  • 1,776
  • 20
  • 21
  • (corrected) It's not about 'answering' CCS; in 1.3 CCS is ignored by the protocol, but often sent anyway to work-around broken middleboxes; see rfc8446 D.4. However the several Java's I can quickly test DO send it as client, although looking at the source I didn't see where it does so. Exactly what Java are you running? Can you capture the working local-Java case and see if/how it looks different? – dave_thompson_085 Mar 11 '23 at 05:29
  • (added) okay, with some further work I mangaged to test the case with HelloRetry (which before I didn't) and in this case JSSE client does NOT send CCS on second flight (with tweaked Hello) as recommended, it sends it on _third_ flight (with Finished). This at least makes your capture consistent with my results, which is less alarming. The USPS server _could_ be failing for this reason, although it _shouldn't_, because this is _allowed_ by the rfc though not recommended. But it doesn't explain how the local-Java to USPS case succeeds, so I would still like to see that. – dave_thompson_085 Mar 11 '23 at 07:06
  • Thanks @dave_thompson_085, I'll have to revert and run again to capture those logs. – Marquee Apr 25 '23 at 19:48

1 Answers1

0

Turns out that Java 11 and TLS 1.3 have a history (see this and this among others). I ended up restricting the app to only use TLS 1.2 and the issue was resolved. USPS is the only API we're using currently that supports TLS 1.3 and hence why it was the only call failing.

java -jar -Djdk.tls.client.protocols=TLSv1.2 -Dhttps.protocols=TLSv1.2 app.jar

Ultimately, this is not a satisfying answer (and not accepted) because these issues are old (and for older versions of Java 11) and there must be a version that works by this point.

Also, this explanation/solution does not square with the fact that the same docker image works locally but not remotely (the Java / Spring Boot versions would be identical). It also is just kicking the can down the road until TLS 1.3 becomes mandatory. We plan to migrate to Java 17 by EoY so hopefully support is better with that version of Java and associated dependencies.

I will update the answer (or accept a competitor) if/when I find a more satisfactory solution.

Marquee
  • 1,776
  • 20
  • 21