How to get the size of an InputStream?

Question

I'm having an InputStream from a ProcessBuilder that acutally reads the stdout stream.

Question: how can I know the size of that inmemory InputStream, so I can write it to a HttpResponse http header?

InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);

OutputStream out = response.getOutputStream();
int bytes;
while ((bytes = br.read()) != -1) {
    out.write(bytes);
}

//how can I know the size of the inmemory stream/file written?
//response.setContentLength((int) pdfFile.length());

You won't know until you have read it all. The output could be endless. — Peter Lawrey, Feb 19 '16 at 09:17
You are better off returning this as a stream to the browser. — Peter Lawrey, Feb 19 '16 at 09:18
Refer this answer : http://stackoverflow.com/questions/11746978/java-inputstream-size — Jitin Kodian, Feb 19 '16 at 09:20
Wait; what tells you that the content is text at all? Also, you don't even specify the encoding on your reader (edit: from the comments it seems that this is not text) — fge, Feb 19 '16 at 09:21
@PeterLawrey could you give an example on how to return this as a stream? — membersound, Feb 19 '16 at 11:15
@membersound how you do it depends on your library e.g. https://dzone.com/articles/jax-rs-streaming-response Do you know you need to set a length? — Peter Lawrey, Feb 19 '16 at 17:20

score 7 · Answer 1 · answered Feb 19 '16 at 09:22

7

There is no such thing as the size of an input stream. Consider a program which never exits, or a socket peer which never stops sending. And you don't need to know to write it to an HttpResponse header. The Content-length is managed automatically for you.

answered Feb 19 '16 at 09:22

user207421

305,947
44
307
483

@BalusC Better use a temp file why? – user207421 Feb 19 '16 at 09:57
@BalusC 'Memory friendly' how? According to your comment, the memory usage doesn't exceed 'the buffer'. Using a temp file only adds latency: it doesn't really save memory. – user207421 Feb 19 '16 at 10:03
@BalusC You haven't answered my question. – user207421 Feb 20 '16 at 01:28
@BalusC But he doesn't need to set the content-length. `HttpResponse` will do that for him, and chunk it too if necessary. – user207421 Feb 26 '16 at 02:06
@BalusC Adding latency just so that progress can be reported isn't much of an idea. The real progress is the progress of the `Process` being executed, and that remains hidden from the client. – user207421 Jan 09 '17 at 06:13
@downvoter Which is it? There is such a thing as the size of an input stream? Content-length isn't managed automatically for you? Please. Don't kid yourself. – user207421 Nov 30 '17 at 02:21
@membersound EJP is right, providing a stream to the response without explicit content length makes the HTTP response *chunked* effectively streaming data back to the client without limitations. With streams you should not assume you know the stream size upfront – gusto2 Feb 27 '18 at 06:55

lance-java · Answer 2 · 2017-11-29T09:21:07.517

2

If you really want to set the content length header, you'll need to read the entire stream before writing to the response OutputStream

ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] bytes = new byte[1024];
int count;
while ((count = in.read(bytes)) > 0) {
    out.write(bytes, 0, count);
}
response.setContentLength(out.size();
out.writeTo(response.getOutputStream());

Note: With this approach you've now read the entire stream into memory, this will have an impact on available memory and likely won't scale well.

edited Nov 29 '17 at 09:21

answered Feb 19 '16 at 09:33

lance-java

25,497
4
59
101

It is all pointless, as Java already manages the Content-length for you. – user207421 Nov 29 '17 at 09:27
1

huh? no it doesn't? Headers (including content-length) are sent before writing to the outputstream. If you write a single byte to the outputstream, you can't set a header at that point. So, the stream will be sent without a content-length header, meaning the client can NOT display a percentage complete whilst the stream is being received – lance-java Nov 29 '17 at 09:32
Huh, yes it does. Try it some time, like I did 20 years ago, and again last month. – user207421 Nov 30 '17 at 02:22
1

Absolute nonsense. Http protocol is headers first, then response body. Sure you can write to the response OutputStream without setting content length header. But if you set it you set it there's benefits. Or maybe you've got a filter in the chain doing it for you? Eg gzip filter – lance-java Nov 30 '17 at 05:06
So in other words you still haven't tried it. Don't waste any more of my time until you have. You will get a surprise. – user207421 Nov 30 '17 at 09:47
huh? of course I've tried it. If you don't set the content-length header on the response, the content-length header will not be set. If in your application you are not setting the content-length header and you can see a content-length header on the response in the client then something is doing it for you. This is NOT default behaviour and you likely have a filter configured which is doing it for you. You say java manages the content-length header.. which framework? There's 100s of web frameworks to choose from in java. Saying java does this for you is nonsense – lance-java Nov 30 '17 at 10:04
The only two frameworks under discussion here are (1) `java.net.HttpURLConnection`, which behaves as I described, and which is in the JRE, and (2) `HttpServletResponse`, ditto, which is in the Servlet API. The former outputs a Content-length header automatically, unless you're using chunked transfer mode, and the latter *uses* chunked transfer mode by default. In neither case do they need any assistance from the application programmer. I have twenty years' worth of working code that demonstrates it. – user207421 Dec 01 '17 at 05:33
Most JavaWebServers buffer ServletOutputStreams up to a certain length. If you close the stream before the bufferlength is reached it will write a content length header and the buffered bytes. If you write more bytes it might use a chunked encoding or more likely an indefinite body which requires the connection to be closed after serving it. This should be avoided. – eckes Feb 20 '18 at 09:37
For tomcat it is for example documented here in doGet() https://tomcat.apache.org/tomcat-8.0-doc/servletapi/javax/servlet/http/HttpServlet.html#doGet(javax.servlet.http.HttpServletRequest,%20javax.servlet.http.HttpServletResponse) – eckes Feb 20 '18 at 10:09
1

"The content length is automatically set if the entire response fits inside the response buffer." I stand corrected – lance-java Feb 20 '18 at 10:59

score 2 · Answer 3 · answered Feb 19 '16 at 09:33

2

Try this

    InputStream is = process.getInputStream();
    ByteArrayOutputStream os = new ByteArrayOutputStream();
    int b;
    while ((b = is.read()) != -1)
        os.write(b);
    response.setContentLength(os.size());
    response.getOutputStream().write(os.toByteArray());

answered Feb 19 '16 at 09:33

It is never necesary to set the content-length in the Java JDK, and using a `ByteArrayOutputStream` here is just a waste of time and space. And doesn't answer the question. – user207421 Nov 29 '17 at 09:24
@user207421 That's just not true. I need to handle content-length myself and this is exactly what I needed. – Lerk May 28 '19 at 13:11
@Lerk You think you need that why? – user207421 Aug 15 '21 at 23:37
@user207421 I had some library that required this. But I don't remember which one after two years, sorry :( – Lerk Aug 19 '21 at 11:33

score 1 · Answer 4 · answered Feb 27 '18 at 06:29

import org.apache.commons.io.IOUtils;

   byte[] bytes = IOUtils.toByteArray(inputStream);
   log.message("bytes .lenght "+bytes.length);

            if (bytes.length > 400000)
//some byte range limit`enter code  can add any byte range
              {
                throw new Exception("File Size is larger than 40 MB ..");
              }

score 0 · Answer 5 · answered Feb 19 '16 at 10:02

An InputStream inherently doesn't have a size. It could conceivably keep delivering bytes forever. Or the producing end could end the stream without warning.

If you must find out the length, then you have to read to the end, counting the bytes, and report the length when you finish.

You're worrying about HTTP's Content-length header, and you've got a point. The fact is that the original version of HTTP was not designed for large, dynamically generated content. The protocol inherently expects you to know the size of the content before you start writing it - yet how is that possible if it's (for example) an ongoing chat, or the output of a video camera?

The solution is HTTP's chunked transfer encoding. Here you don't set a Content-Length header. You set Transfer-Encoding: chunked, then write the content as chunks, each of which has a size header.

The HTTP RFC has details one this, or https://en.wikipedia.org/wiki/Chunked_transfer_encoding is slightly more friendly.

However most HTTP APIs hide this detail from you. Unless you are developing a web library from scratch (perhaps for academic reasons), you shouldn't have to think about Content-Length or Transfer-Encoding.

How to get the size of an InputStream?

5 Answers5

Linked