21

I am using HttpURLConnection to write files, some of which are quite large, to a server. 

final HttpURLConnection conn = (HttpURLConnection) url.openConnection();

A while back I had issues writing objects 1 GB or more. I fixed that by setting it to stream a more manageable chunk size.

final int bufferSize = 1024 * 1024;
[...]
conn.setChunkedStreamingMode(bufferSize);

It had then been running fine on my laptop, but on other machines it was crashing. On investigation, I found that the cause was an out of memory error that occurred when writing to the output stream.

final OutputStream out = conn.getOutputStream();
final long bytesWritten = IOUtils.copyLarge(in, out);

Inside the copyLarge routine I found that that it was able to do 262145 iterations of 4096 bytes, failing when it tried to cross the 1 GB line. Allocating more memory to the java application seemed to prevent those crashes, but I thought that should be unnecessary. If it is writing chunks of 1 MB, then either it should fail with far fewer iterations or repeatedly write 1 MB without issue.

UPDATE: Turns out the line setting the ChunkedStreamingMode wasn't actually being called on some machines. If you don't set fixed/chunked streaming mode, HttpURLConnection will just send everything to a PosterOutputStream/ByteArrayOutputStream.

Arjan
  • 22,808
  • 11
  • 61
  • 71
WindowsWeenie
  • 389
  • 1
  • 5
  • 11
  • 4
    Have you verified that it's using `Transfer Encoding: chunked`? Otherwise it will need to buffer everything to compute the `content-length`. Something like Wireshark might help you diagnose if the data is being sent properly. – Jonathan Apr 20 '11 at 16:48
  • Another idea is be that the client caches the data after sending for the case that the server sends back some error code, to be able to retry. – Paŭlo Ebermann Apr 20 '11 at 22:46
  • 1
    Interesting, what machines/JRE versions were it? Did you fix it? If so, how exactly? – BalusC Apr 22 '11 at 19:25
  • 5
    Could you post an answer describing the solution? – Paŭlo Ebermann Jun 13 '11 at 20:40
  • Any solution this? What i understand it, you are doing server to server data transfer (File upload), right? And i think even if we run such operation in new thread, it will be blocking as it's an IO. Have you thought of `FTP protocol/ Apache HttpClient/ File upload from HTML Form`? – Parth Aug 23 '11 at 09:28
  • 2
    Can someone close this question? Otherwise it'll keep appearing in the 'unanswered' section, even though the asker answered his own question. – Jord Sonneveld Aug 29 '11 at 17:34
  • 2
    Please post your solution as an answer, and then accept it. :) – Unsigned Sep 03 '11 at 04:22

0 Answers0