2

I am trying to pull page source in java with proxy.

I have used httpclient, httpunit and HttpURLConnection, all of this taking 5 sec when i goes for page source without proxy but I uses proxy then its taking 12-15 sec for a single page.

Is there any way to increase speed or another class which is providing fast page source.

skaffman
  • 398,947
  • 96
  • 818
  • 769
Khoyendra Pande
  • 1,627
  • 5
  • 25
  • 42

1 Answers1

1

You need to identify where the problem is first: is it the case that the server you are pulling the pages from is really slow or is it the fact that the proxy you use is really slow (or both)? If you use the likes of wget (or probably even curl) in the command line that should show you statistics about the download time of the page -- through proxy and bypassing the proxy. If those times are the same with what your code shows then there's nothing you can optimise. If however, you see discrepancies then you can look at using other ways of pulling the pages in your code. As a general rule of thumb:

  • use Java NIO if you're dealing with lots of connections
  • use normal Java IO if you're dealing with large transfers to only a handful of hosts/servers
Liv
  • 6,006
  • 1
  • 22
  • 29