Apache HttpClient not receiving entire response

Question

Update: If I use System.out.println(EntityUtils.toString(response.getEntity())); the output is what appears to be the missing lines of HTML (including the closing body and html tags). However, printing to a file still only gives me the first 2000 odd lines missing the last 1000.

I am using the following code to perform a http post request:

public static String Post(CloseableHttpClient httpClient, String url, Header[] headers,
            List<NameValuePair> data, HttpClientContext context) throws IOException
{
    HttpPost httpPost = new HttpPost(url);
    httpPost.setEntity(new UrlEncodedFormEntity(data));
    httpPost.setHeaders(headers);
    CloseableHttpResponse response = httpClient.execute(httpPost, context);

    if (response.getEntity() == null)
        throw new NullPointerException("Unable to get html for: " + url);

    // Get the data then close the response object
    String responseData = EntityUtils.toString(response.getEntity());
    EntityUtils.consume(response.getEntity());
    response.close();

    return responseData;
}

However I am not receiving the full response entity. I am missing about 1000 lines of html (including the closing body and html tags. I think this is because the data is being sent in chunks although I am not entirely sure.

Here are the response headers:

Cache-Control:max-age=0, no-cache, no-store
Connection:Transfer-Encoding
Connection:keep-alive
Content-Encoding:gzip
Content-Type:text/html; charset=utf-8
Date:Sat, 04 Jul 2015 15:14:58 GMT
Expires:Sat, 04 Jul 2015 15:14:58 GMT
Pragma:no-cache
Server:Microsoft-IIS/7.5
Transfer-Encoding:chunked
Vary:User-Agent
Vary:Accept-Encoding
X-Content-Type-Options:nosniff
X-Frame-Options:SAMEORIGIN

How can I ensure that I receive the full response entity?

@WandMaker I don't think so, see http://stackoverflow.com/questions/15969037/why-did-the-author-use-entityutils-consumehttpentity — james, Jul 04 '15 at 16:14
@Antoniossss you were right. Originally I was using a single line creating a printwriter and then just calling print. However, using it properly (calling flush() and close()) ensured that all of the html was written to the file. Unfortunately I am still missing html so it means my request is wrong :/ — james, Jul 04 '15 at 16:26
turns out a cookie was being set that I need to include in my post (for those interested) — james, Jul 04 '15 at 18:03

score 1 · Accepted Answer · answered Jul 04 '15 at 16:42

1

To gather all the essence of the comments. There is nothing wrong with your code here - using EntityUtils is the recomended way to deal with all kinds of responses. You have error in code that stores your response to the file.

answered Jul 04 '15 at 16:42

Antoniossss

31,590
6
57
99

score 1 · Answer 2 · answered May 07 '18 at 08:28

I've got a similar problem, and solved it by ensuring lcosing the connection like :

} finally {
        try {
            EntityUtils.consume(entity);

            try {
                response.getOutputStream().flush();
            } catch (IOException e) {
                logger.warn("Error while flushing the response output connection. It will ensure to close the connection.", e);
            }

            if (null != httpResponse) {
                httpResponse.close();
            }
        } catch (IOException ignore) {
        }
    }

Or event better with try-resources :

try(CloseableHttpResponse response = httpClient.execute(httpPost, context)){ 
  if (response.getEntity() == null){
    throw new NullPointerException("Unable to get html for: " + url);
  }
  String responseData = EntityUtils.toString(response.getEntity());
  EntityUtils.consume(response.getEntity());
}

Apache HttpClient not receiving entire response

2 Answers2