0

I am using Apache HttpClient 4.5.1 to download a ".tgz" file and I am running my client program on Windows.

When I run, I get the file downloaded with a size 4,423,680 (and it can't open because of the wrong size). But the actual size of the file is 4,414,136 (using "wget").

The problem goes away when I use DefaultHttpClient (deprecated) instead of CloseableHttpClient (all other code being the same).

Code that created the problem: 

import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
CloseableHttpClient httpClient = HttpClients.createDefault();
(or)
CloseableHttpClient httpClient = HttpClientBuilder.create().build();

// the following code is the same in both cases 
HttpGet httpGet = new HttpGet(url);
HttpResponse response = httpClient.execute(httpGet);
try {
   BufferedInputStream bis = new BufferedInputStream(entity.getContent());
   BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(new File(filePath)));
   int inByte;
   while ((inByte = bis.read()) != -1 ) {
                bos.write(inByte);
   }
}
....

Code that resolved the problem: 

 import org.apache.http.impl.client.DefaultHttpClient;
 ...
 HttpClient httpClient = new DefaultHttpClient();
 ... <same code>....

DefaultHttpClient is deprecated, but it seems to work fine.
Don't understand what is wrong with the CloseableHttpClient.

skuniset
  • 1
  • 1
  • (1) Flush the buffer. (2) No need to buffer content stream. It is already optimized for read operations by HttpClient. (3) There are much better ways to write out content stream to a file (4) See Ricard Nàcher Roig's comments – ok2c Feb 13 '16 at 11:13
  • @oleg, How does your comment address my question? I am not looking for suggestions for optimizations here. I have clearly mentioned what is working and what is not working and provided the code chunck in both cases. I am already flushing the buffer and since it is not relevant I didn't show it (see the "...." at the end to indicate that more code follows). I just want to understand why the CloseableHttpClient doesn't work, though all the other code is exactly the same, except the first line that creates the httpClient. – skuniset Feb 16 '16 at 23:17
  • Please excuse my bluntness here, but given the overall quality of your code snippet I cannot help thinking it is more likely to be at fault here than HttpClient. – ok2c Feb 17 '16 at 13:25

2 Answers2

1

I would like to give you several tips.

Fist, I think that you need to close the response, so use a try-with-resources block or add a finally block to the try-catch block 

try(HttpResponse response = httpClient.execute(httpGet)) {
....

or 

finally
{
  response.close();
}

Maybe you already closed it and you didn't copy it into the sample. Please, forgive me if it is the case.

Second. Every time I got a similar behaviour was because the output stream was not flushed and closed. I mean that you need to flush and close the bos stream at the end of your code.

Final note: This was just a comment. Maybe that does not solve your issue. I am sorry but I can't understand why the usage of the other Http Client does not reproduce the same issue.

Ricard Nàcher Roig
  • 1,271
  • 12
  • 14
  • thanks for your comment, but its not related to the problem that I was talking about. I have tried all those things - like flushing, closing properly, with/without buffering, etc. As I said, the ONLY difference between working & non-working is that when I use DefaultHttpClient (which is decrecated) it works, but when I use the new API CloseableHttpClient it doesn't work. – skuniset Feb 16 '16 at 01:16
1

Please try out this code snippet and see if you still get a different file size. 

CloseableHttpClient httpClient = HttpClientBuilder.create()
        .disableContentCompression()
        .build();
HttpGet get = new HttpGet(url);
try (CloseableHttpResponse response = httpClient.execute(get)) {
    HttpEntity entity = response.getEntity();
    if (entity != null) {
        try (FileOutputStream outputStream = new FileOutputStream(filePath)) {
            entity.writeTo(outputStream);
        }
    }
}
ok2c
  • 26,450
  • 5
  • 63
  • 71
  • Thanks that works i.e using "disableContentCompression()" makes the difference. I have tried exactly the same code (one of the various combinations I tried in my earlier attempts) without using "disableContentCompression()" and that didn't work. I was thinking that there are differences in the implementation of "default" client between old API (DefaultHttpClient) and new API in the sense that they are assuming different default values. Anyway, it looks like a bug in the new HTTPClient i.e., even if I enable compression on a file, it shouldn't get a wrong size. Where can I file a bug on this? – skuniset Feb 18 '16 at 00:03
  • What makes you think this is a bug in HttpClient? It looks like the server sends garbage if the client signals that it can accept compressed content. – ok2c Feb 18 '16 at 09:36