4

I have the following code which produces an OutOfMemory exception:

byte[] buf = new byte[10240];
int len = 0;
DataOutputStream dataOS = new DataOutputStream(conn.getOutputStream());
while ((len = _inputStream.read(buf)) > 0) {
    System.out.println("len : " + len);
    System.out.println("Going to write buf into dataOS : " + buf.length);
    dataOS.write(buf, 0, len);
    System.out.println("dataOS.size() : " + dataOS.size());
}
_inputStream.close();

The following is the last few lines from the debug output:

len : 10240
Going to write buf into dataOS : 10240
dataOS.size() : 342804702
len : 10240
Going to write buf into dataOS : 10240
dataOS.size() : 342814942
len : 10240
Going to write buf into dataOS : 10240

The exception occurs when I try to write into the dataOS beyond 342814942.

Could someone please help me handle this?

Thanks!

Ram
  • 1,097
  • 2
  • 12
  • 24
  • 1
    Like sharakan says, you should flush your buffer regularly, but if you really don't want to flush, increase the allocation of memory to your JVM (-Xmx parameter on starting Java). – Renato Apr 24 '12 at 01:32

2 Answers2

13

It has nothing to do with the DataOutputStream (which maintains no data whatsoever) and everything to do with your underlying stream conn.getOutputStream(). now, you haven't show the relevant code there, but i'm going to guess "conn" is an instance of HttpURLConnection. i believe the outputstream for HttpURLConnection buffers the output so that it can determine the output length (unless you set it explicitly). if you know the output length, you could set that directly using HttpURLConnection.setFixedLengthStreamingMode, or you could call HttpURLConnection.setChunkedStreamMode to allow the data to be sent in chunks instead of buffered entirely in memory.

in the future, when you encounter an OOME you should always generate a heap dump and open that in a memory profiler. that will almost always show you immediately where the problem is (e.g. in the underlying stream, not the DataOutputStream).

jtahlborn
  • 52,909
  • 5
  • 76
  • 118
  • jtahlborn: Thanks a lot. The problem was indeed to do with conn.getOutputStream(). And yes, it is a HttpURLConnection. Apologies for not being specific. I tried setting setFixedLengthStreamingMode to the file length and it seems to be going beyond the previous size. Thank you. I am using the dataOS to upload data to a server. Hence, when I use setChunkedStreamMode, do I have to handle upload in a different way? Right now I call conn.getResponseCode() to upload. Please suggest. – Ram Apr 24 '12 at 01:54
  • 2
    @Ram - for one thing, you have no need of the DataOutputStream (just use the conn output stream directly). if you use the chunked streaming, _you_ don't need to do anything different on your end. the server on the other end needs to support chunked encoding (but most do). – jtahlborn Apr 24 '12 at 02:03
1

Depending on the implementation of OutputStream, writing may buffer the content in memory, and not actually send it. If you buffer enough content, you will of course run out of memory.

At some interval, perhaps each time through your loop, you should call OutputStream.flush(). If there is a buffer in use, that will clear it.

sharakan
  • 6,821
  • 1
  • 34
  • 61
  • 2
    pretty much every stream implementation which buffers data will _automatically_ flush the data when the internal buffer is full. – jtahlborn Apr 24 '12 at 01:36
  • @jtahlborn That wouldn't surprise me, but as far as I know that's not part of the contract. Perhaps you can give an example? – sharakan Apr 24 '12 at 01:44
  • every stream implementation in java, except maybe the http output stream when it is attempting to determine the output length. – jtahlborn Apr 24 '12 at 01:47