0

I need to write a lib, which does many remote HTTP calls to fetch a content. I could do it as described here, but is there a better way (in sense of performance) how to do it? If I do it as described in example I do always create an URL object, which is parsed and so on - so I think it kind of overhead: base URL is always the same, just single parameter is always different, e.g.:

http://www.example.com/?param=value1
http://www.example.com/?param=value2
http://www.example.com/?param=value3
and so on...

What would be more faster/better way?

Laimoncijus
  • 8,615
  • 10
  • 58
  • 81
  • 1
    Unless you're on the fastest internet connection in the universe, parsing a URL is nothing compared to waiting for a response from the server. – Brendan Long Oct 28 '11 at 17:04
  • @Brendan Long: good point, but still when I do tens of thousands of calls like that - I could save some computing power on things, which I don't really need... – Laimoncijus Oct 28 '11 at 17:06
  • the URL parsing would indeed be trivial. The most performant way to do this is to figure out how you can make as few http calls as possible. – matt b Oct 28 '11 at 17:12

2 Answers2

3

Assuming each URL has the potential to return a different resource, that's all there is. I would put each URL in it's own thread so you don't block for network issues.

But make sure you don't create too many threads. If you have more than, say, 10, I would say put the URLs in a queue and have each thread process the next one in the queue when complete so you don't gum up your system with threads.

Thom
  • 14,013
  • 25
  • 105
  • 185
  • Or even faster would be using [asynchronous requests](http://stackoverflow.com/questions/1014528/asynchronous-http-client-for-java), but it's generally harder. – Brendan Long Oct 28 '11 at 17:06
1

Use HttpClient and normal form semantics; you may just be able to overwrite the parameter value.

OTOH, compared to network speed, I find it difficult to believe that URL parse time will be the gating issue. Other than that, you could spin up multiple request threads.

Dave Newton
  • 158,873
  • 26
  • 254
  • 302
  • My idea was not to save time (as I could really use threads), but to save some computing power and skip URL parsing as I know it to be valid and with every different parameter it will be sill OK, so in my case no need to validate/parse it on every call... – Laimoncijus Oct 28 '11 at 17:12
  • 1
    @Laimoncijus I understand, but you're saving essentially nothing, that's all I'm saying. – Dave Newton Oct 28 '11 at 17:13
  • @Dave Newton: good to know that it doesn't cost anything; means I don't have to really bother about it – Laimoncijus Oct 28 '11 at 17:16
  • @Laimoncijus *Essentially* nothing. Everything has a cost; you just have to determine if the ROI is worth the effort. – Dave Newton Oct 28 '11 at 17:17