Intermittent SSL/TLS error on file upload

Question

I have some difficulties resolving an SSL/TLS error while consuming a REST web service.

The REST web service is developed with the Spring Framework, using Spring Boot.

The client is a .NET application, using RestSharp for the REST web service consumption.

Everything worked fine until one client had an error during a file upload.

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

I was still using the embedded Tomcat v8.5.5 at this time. On my Spring Boot application I had the following errors:

java.lang.NullPointerException: null
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.getSslSupport(NioEndpoint.java:1329) ~[tomcat-embed-core-8.5.5.jar!/:8.5.5]
    // Etc.

java.lang.NullPointerException: null
    at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:182) ~[tomcat-embed-core-8.5.5.jar!/:8.5.5]
    // Etc.

After some researches, I found this topic about similar errors related to the embedded Tomcat v8.5.5. Following their advices, I upgraded the embedded Tomcat to v8.5.13.

Nice, no more NPE from apache Tomcat. So, my .NET application should work... or not.

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Still the same error. Since I had no Tomcat error anymore, I changed the logging level of org.apache to DEBUG. And I found some new errors at the DEBUG level:

2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.a.tomcat.util.net.SocketWrapperBase    : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@3678fe26:org.apache.tomcat.util.net.SecureNioChannel@63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Read from buffer: [0]
2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header

java.io.EOFException: null
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1242) ~[tomcat-embed-core-8.5.13.jar!/:8.5.13]
    // Etc.

2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.apache.coyote.http11.Http11Processor   : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@3678fe26:org.apache.tomcat.util.net.SecureNioChannel@63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Status in: [OPEN_READ], State out: [CLOSED]
2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] org.apache.tomcat.util.net.NioEndpoint   : Failed to close socket

java.nio.channels.ClosedChannelException: null
    at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source) ~[na:1.8.0_91]
    // Etc.

2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.apache.tomcat.util.threads.LimitLatch  : Counting down[https-jsse-nio-443-exec-4] latch=1

2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.a.tomcat.util.net.SocketWrapperBase    : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@58361212:org.apache.tomcat.util.net.SecureNioChannel@63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Read from buffer: [0]
2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header

java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
    at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:604) ~[tomcat-embed-core-8.5.13.jar!/:8.5.13]
    // Etc.

2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.apache.coyote.http11.Http11Processor   : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@58361212:org.apache.tomcat.util.net.SecureNioChannel@63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Status in: [OPEN_READ], State out: [CLOSED]
2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] org.apache.tomcat.util.net.NioEndpoint   : Failed to close socket

java.nio.channels.ClosedChannelException: null
    at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source) ~[na:1.8.0_91]
    // Etc.

2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.apache.tomcat.util.threads.LimitLatch  : Counting down[https-jsse-nio-443-exec-9] latch=1

For what I understand, the client seems to successfully connect to the server but proceed to interrupt the connection during file transfer because of an SSL/TLS error, resulting in those ClosedChannelException server-side.

Am I using RestSharp wrongly?

RestRequest request = new RestRequest("path/to/uri", Method.PUT);

request.AddQueryParameter("myParam", myParam.ToString());
request.AddFile("myFile", "path/to/file", "multipart");

IRestResponse<MyResponse> response = myRestClient.Execute<MyResponse>(request);

Spring Boot code for this request:

@RequestMapping(method = RequestMethod.PUT, consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public MyResponse replace(@RequestParam Integer myParam, @RequestParam MultipartFile myFile) {
    // Etc.

The PUT method for multipart is not supported by default in Spring but I created my own MultipartResolver to handle it so the problem is not here.

It looks like the problem only happens in one place, and only during file transfer. All other places work fine, but there are still the EOFException and ClosedChannelException server-side, even when it works client-side.

Another obscure thing... it only happens sometimes. Sometimes the upload works, sometimes not.

I am not an SSL expert so I wonder what could do that.

Something not synchronized between the client and the server?
An interruption between the client and the server, resulting the ClosedChannelException?

I am open to all suggestions and will continue to investigate, and hopefully find something...

EDIT 2017-05-11

Precision from comments:

The .NET application is built with the Microsoft Windows .NET Framework (not Mono).
The SSL certificate (server-side) is a complete chain in a PKCS#12 file.

Information retrieved since I posted the question:

The bug only happens if the .NET application is open for a certain amount of time; it will not occur if opened and closed in less than a minute.
Host OS: Windows 10 Pro x64... but System.Environment.OSVersion.VersionString gives Windows NT 6.2.9200.0. According to Wikipedia this kernel version correspond to Windows 8 or Server 2012, not 10. EDIT Ok, this is a known behavior. See here and here. So this has nothing to do with the error.
Trying to reproduce the bug without success on my side. I suspect more and more a bad configuration on the host machine or network.
As seen on the code above, the RestClient object is instantiated only once and used everywhere in the application. Could it be an issue?

Investigation continues.

Is the entire chain of certification embedded in the certifcate your server is presenting to the clients ? Clients *will* complain about a certificate if they can't validate the entire chain. The thing is, even if they have the root certificate of the entity who issued your certificate, if they miss only one intermediate certificate, the chain of trust will be broken. — Irwene, May 10 '17 at 15:12
@Sidewinder94 Yes it is. For all other operations, no SSL issue. And sometimes the file upload works without SSL issue. — kagmole, May 10 '17 at 15:42

Intermittent SSL/TLS error on file upload

0 Answers0