3

I am trying to peek at an input stream contents from HttpClient, up to 64k bytes.

The stream comes from an HttpGet, nothing unusual about it:

HttpGet requestGet = new HttpGet(encodedUrl); HttpResponse httpResponse = httpClient.execute(requestGet); int status = httpResponse.getStatusLine().getStatusCode(); if (status == HttpStatus.SC_OK) { return httpResponse.getEntity().getContent(); }

The input stream it returns is of type org.apache.http.conn.EofSensorInputStream

Our use-case is such that we need to "peek" at the first (up to 64k) bytes of the input stream. I use an algorithm described here How do I peek at the first two bytes in an InputStream?

PushbackInputStream pis = new PushbackInputStream(inputStream, DEFAULT_PEEK_BUFFER_SIZE); byte [] peekBytes = new byte[DEFAULT_PEEK_BUFFER_SIZE]; int read = pis.read(peekBytes); if (read < DEFAULT_PEEK_BUFFER_SIZE) { byte[] trimmed = new byte[read]; System.arraycopy(peekBytes, 0, trimmed, 0, read); peekBytes = trimmed; } pis.unread(peekBytes);

When I use a ByteArrayInputStream, this works with no problem.

The Issue: When using the org.apache.http.conn.EofSensorInputStream I only get a small number of bytes at the beginning of the stream. usually around 400 bytes. When I expected up to 64k bytes.

I also tried using a BufferedInputStream where I read up to the first 64k bytes then call a .reset() but that doesn't work either. Same issue.

Why might this be? I do not think anything is closing the stream because if you call IOUtils.toString(inputStream) I do get all the content.

Nicholas DiPiazza
  • 10,029
  • 11
  • 83
  • 152

1 Answers1

2

See InputStream#read(byte[] b,int off,int len) contract:

Reads up to len bytes of data from the input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read. The number of bytes actually read is returned as an integer

Instead of using this method, use IOUtils.read which reads until you get the number of bytes you requested in a loop.

Nicholas DiPiazza
  • 10,029
  • 11
  • 83
  • 152
ok2c
  • 26,450
  • 5
  • 63
  • 71
  • aw dang it! any idea how to make it return more? do i just use the offset and make another request? are you sure that's not just saying that it will return less bytes if it doesn't have any more left? – Nicholas DiPiazza Jun 11 '17 at 14:49
  • Yes, I am quite sure. You just need to keep on reading in a loop until the desired of amount of data has been received or end of stream is reached. – ok2c Jun 12 '17 at 07:19
  • This did not work. `available()` returns 0. It thinks the stream is empty so adding the loop that you suggested did nothing. – Nicholas DiPiazza Jun 12 '17 at 15:17
  • instead of using `read` from the InputStream, using `IOUtils.read()` (which has the loop we were discussing) works!. so disregard my previous comment. – Nicholas DiPiazza Jun 12 '17 at 15:49