25

I'm trying to upload a file to a web-service that is protected with SSL and requires a client-side certificate (signed by an in-house CA). The communication to the web-service works well (Downloading files, Querying, running commands and performing all sorts of POSTs works well as expected), except for uploading files.

When uploading files I get an SSLException (javax.net.ssl.SSLException) that says "Write error: ssl=0x5fe209c0: I/O error during system call, Connection reset by peer".

I have created a duplicate server and removed the SSL and Client-Certificate requirements, and tried to upload over 'vanilla' HTTP, and it works perfectly.

I've tried using setFixedLengthStreamingMode(int) and setChunkedStreamingMode(int) without success. When using them, the exception is thrown from the write method, and when not using any of them, the same exception is thrown from the call to getResponseCode().

I couldn't find anything about the error in the server's EventVwr.

Our other client (iOS client) is able to upload files there, so it must be something that I do - but I can't figure out what.

I'm not sure how to debug this issue further.

Please help.

Edit 1

We've done a lot of debugging efforts, and found that:

  • Small files are uploaded as expected (44kb is the size of the largest file that was uploaded successfully, and it uploaded in ~1200ms).
  • An 46kb file failed to upload. The failure took ~2 minutes (134120ms).

Edit 2

After what you'll read in the remarks, now I got Fiddler to play nice (Thanks to this question). Fiddler got the file, but did not succeed in sending it. The requests (raw) looks like:

POST https://192.168.2.2/rest/transfer/strong/Upload/Full?Path=%5C20140807_113255_20.jpg&Root=2 HTTP/1.1
SessionToken: 1234 // We use this for session management
FileMetadata: {"FileSize":"1315496","FileName":"GrumpyCat.jpg"}
Connection: Keep-Alive
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.1.1; GT-N7100 Build/JRO03C)
Host: 192.168.2.2
Accept-Encoding: gzip
Content-Type: application/x-www-form-urlencoded
Content-Length: 1315496

;odiao;awriorijgoeijoeirj;oedfrvgerg... // The image

Fiddler's response (also RAW) was:

HTTP/1.1 504 Fiddler - Send Failure
Date: Wed, 20 Aug 2014 17:40:29 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Timestamp: 20:40:29.420

[Fiddler] ResendRequest() failed: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. < An existing connection was forcibly closed by the remote host

Also, We've added WCF's 'MessageLogging' and verbose 'Tracing'. MessageLogging don't show any hint of the message (probably dropped before turning into a message), but the trace showed this: The WCF Trace as seen from the SvcTraceViewer

Now, before you say "ahhh, this is a server problem", keep in mind that 44kb files succeed in uploading, and our iOS app also is able to upload files successfully.

This is the call stack from the exception that the client gets:

E/RestClientUploader(3196): javax.net.ssl.SSLException: Write error: ssl=0x5d94b8b0: I/O error during system call, Connection reset by peer
E/RestClientUploader(3196):     at org.apache.harmony.xnet.provider.jsse.NativeCrypto.SSL_write(Native Method)
E/RestClientUploader(3196):     at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:693)
E/RestClientUploader(3196):     at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:231)
E/RestClientUploader(3196):     at libcore.net.http.ChunkedOutputStream.writeBufferedChunkToSocket(ChunkedOutputStream.java:129)
E/RestClientUploader(3196):     at libcore.net.http.ChunkedOutputStream.write(ChunkedOutputStream.java:77)
E/RestClientUploader(3196):     at java.io.DataOutputStream.write(DataOutputStream.java:98)
E/RestClientUploader(3196):     at com.varonis.datanywhere.communication.RestClientUploader.uploadFileToServer(RestClientUploader.java:151)
E/RestClientUploader(3196):     at com.varonis.datanywhere.communication.RestClientUploader.uploadFullFile(RestClientUploader.java:67)
E/RestClientUploader(3196):     at com.varonis.datanywhere.communication.services.FileUploadService.doUpload(FileUploadService.java:128)
E/RestClientUploader(3196):     at com.varonis.datanywhere.communication.services.FileUploadService.onHandleIntent(FileUploadService.java:98)
E/RestClientUploader(3196):     at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:65)
E/RestClientUploader(3196):     at android.os.Handler.dispatchMessage(Handler.java:99)
E/RestClientUploader(3196):     at android.os.Looper.loop(Looper.java:137)
E/RestClientUploader(3196):     at android.os.HandlerThread.run(HandlerThread.java:60)
Community
  • 1
  • 1
Felix
  • 1,034
  • 1
  • 9
  • 29
  • That is quite odd. Is this a multi-part upload ? (may be a digression). Perhaps the server has already responded to you and yet you are continuing to attempt to write to the server using the same connection. By this time the server has sent back a `RST` and carried on. You can use a web debugging proxy like [Charles](http://www.charlesproxy.com/) to intercept and monitor the HTTPS traffic (requires you to accept charles CA on the client). Monitor the HTTPS response from the server and tell us what that looks like. – Deepak Bala Aug 17 '14 at 16:00
  • @DeepakBala - I tried that, but It **didn't** work. I've set the client to accept all certificates (custom trust manager that trusts everything), but once Fiddler (Same as charles AFAIK) is set to be the proxy, the connection fails (SocketException that says "Socket is closed" during the checkOpen method (in startHandshake) of OpenSSLSocketImpl.java (although I see the tunneling in Fiddler). – Felix Aug 18 '14 at 12:05
  • 1
    So what does Fiddler report in terms of traffic to the server ? What happens when it proxies your request and what is the response given by the server ? – Deepak Bala Aug 18 '14 at 12:25
  • @DeepakBala - Funny thing that Fiddler is. I've managed to get the Fiddler to show SSL sessions, but only without using client certificate (and in these cases, the Upload works as expected). When using a client-side certificate and introducing fiddler (and having our client trust all certificates), I can see 2 Tunneling sessions in Fiddler, but the communication breaks down and the login (prerequisite to upload) fails as I described in the previous comment. – Felix Aug 20 '14 at 08:04
  • Can you post whatever you see in Fiddler along with code that initiates the client side transfer ? – Deepak Bala Aug 20 '14 at 08:46
  • So the service implementation is in WCF ? I have little experience there. If I were to hazard a hunch, I'd say that the binding configuration probably limits the file size to X kb ? Can you check your configuration and compare them with the ones in [this article](http://cgeers.com/2009/08/07/wcf-over-https/) ? Perhaps `maxReceivedMessageSize` is low ? – Deepak Bala Aug 20 '14 at 19:08
  • @DeepakBala - That was one of the first things I've checked (and now re-checked) - But uploading files (larger then 1mb) works great from the iOS client. Not a server configuration problem. – Felix Aug 21 '14 at 09:13
  • I'm afraid my lack of knowledge on WCF stops me from providing any further clues on the problem. It looks like you're on the right track though. Let us know what the solution is when you find it. – Deepak Bala Aug 21 '14 at 13:18
  • 1
    This may be related to this issue: https://code.google.com/p/android/issues/detail?id=61706 – Luis Sep 19 '14 at 03:07
  • @Leco, I don't think it is. While I'm not 200% sure, I can see in WireShark the server sending the FIN after 2 minutes - which tells me that the connection is still alive - and that this FIN is causing the android device's connection to be "reset by peer". – Felix Sep 22 '14 at 14:52
  • You're probably already aware of this, but this seems to be almost exactly the same issue: https://code.google.com/p/android-developer-preview/issues/detail?id=964 – biziclop Oct 17 '14 at 21:04
  • Can you put a breakpoint into `libcore.net.http.ChunkedOutputStream.writeBufferedChunkToSocket` and check what size chunks are sent? – biziclop Oct 17 '14 at 21:25
  • @biziclop: 1. Not exactly the same issue - (A) The developer there got a different exception. (B) His workaround isn't possible because it has a huge impact on our servers (we're fully distributed, and the suggested workaround would have forced us to keep state in our Upload process, which is stateless). We've decided to break the upload process into 2 calls, one with small payload (metadata) and returns a token, and one that takes the token and stream - which doesn't require the Client-Certificate. 2. Can I put a breakpoint there? it's sooo internal to the Android library.. How to do that? – Felix Oct 19 '14 at 10:21
  • try to increase timeout and check – Panther Oct 26 '14 at 07:09
  • @Panther, The timeouts set on the HttpsUrlConnection had no effect. The timeouts on the server behaved as expected, I had to wait longer for that pesky exception. Keep in mind that uploading a 44kb file took a few seconds, so the 2 minutes timer for 64kb should suffice. – Felix Oct 27 '14 at 08:37
  • This may seem like a silly question, but is your SSL certificate expired? I have seen similar behavior (upload failing while other actions function properly) in web apps with expired ssl cert. – Andrew Campbell Oct 28 '14 at 17:32
  • @AndrewCampbell, The certificate is valid (Not expired and not revoked). While I didn't mention it, the certificate is generated and there's a whole flow of "PKCS10 (CSR) --> PK7B --> PKCS12" with the in-house CA, making sure that the certificate is valid (read:Not expired) for at least a month, else we generate a new certificate. – Felix Oct 30 '14 at 08:42
  • This is not a certificate issue, otherwise you could never get started. It seems to be that you're exceeding an upload limit imposed by the server somehow. All it can do when you exceed it is close the connection, which causes 'connection reset by peer'. – user207421 Oct 30 '14 at 11:18

2 Answers2

2

Not an answer, more a work-around, for your reference.

After bashing our heads around this issue, and doing a lot of research, we gave up. We've opened this issue with Google, and implemented the following work around:

In order to upload a file, the app first gets an Upload Token via an endpoint that requires the client-certificate, and afterwards uses this token to upload to an endpoint that doesn't require the client certificate (but still over SSL (Https)).

Yes, it's a minor breach of security, but we had to do it. We've protected it as much as we could.

I promise to update when the Google's ticket will be updated (and hopefully resolved).

halfer
  • 19,824
  • 17
  • 99
  • 186
Felix
  • 1,034
  • 1
  • 9
  • 29
  • The link provided seems to require an account. Are you in a position to update this with any new developments? – halfer Jul 21 '19 at 23:18
1

It's a bit late (as you already implemented a workaround) but this should solve the problem: https://stackoverflow.com/a/9224892/1619545

We're experiencing the same issue, setting the client cert negotiate flag to enabled seems to be the only thing hat helps. Have a look here for a way how to change the flag at the cert binding:

http://help.sap.com/saphelp_smp305svr/helpdata/en/6f/f0a9b6e1c743d48d1e57235d297c1c/content.htm

Community
  • 1
  • 1
AndyB
  • 556
  • 1
  • 9
  • 25