6

I am doing a webcrawler and using threads to download pages.

The first limiting factor to the performance of my program is the bandwidth, I can never download more pages that it can get.

The second thing is what I interested. I am using threads to download many pages at same time, but as I create more threads, more sharing of processor occurs. Is there some metric/way/class of tests to determine what is the ideal number of threads or if after certain number, the performance doesn't change or decrease?

radoh
  • 4,554
  • 5
  • 30
  • 45
Renato Dinhani
  • 35,057
  • 55
  • 139
  • 199
  • 2
    If your application were CPU-bound, then the number is pretty close to the number of CPUs you have on your machine. However, I/O-bound programs (like web crawlers) end up blocking on I/O a lot, so you might be able to have more. Thus, I have to agree with @pst's comment: Run tests, then figure out the ideal number for your environment. There isn't a really good way to detect this in code. (However, you CAN write self-tuning code once you know how to check your "current average" bandwidth.) – Platinum Azure May 19 '11 at 22:20

4 Answers4

0

The ideal number of thread should be close to the number of cores (virtual cores) your hardware provides. This is to avoid thread context switching and thread scheduling. If you're doing heavy IO operations with many blocking reads (your thread blocks on a socket read) I suggest you redesign your code to use non-blocking IO APIs. Typically this will involve one "selector" thread that will monitor the activity of thousands of sockets and a small number of worker threads that will do the processing. If you code is in Java, the APIs are NIO. The only blocking call will be when you call selector.select() and it will only block if there is nothing to be processed on any of the thousands of sockets. Event-driven frameworks such as netty.io use this model and have proven to be very scalable and to best use the hardware resources of the system.

Jean de Lavarene
  • 3,461
  • 1
  • 20
  • 28
0

we've developped a multithreaded parrallel web crawler. Benchmarking troughput is the best way to get ideas on how the beast will handle his job. For a dedicated java server, one thread per core is a base to start, then the I/O comes into play and change.

Performances do decrease after certain number of threads. But it depends on the site you crawl too, on the OS you use, etc. Try to find a site with a merely constant response time to do your first benchmarks (like Google, but take differents services)

With slow websites, higher number of threads tends to compensate i/o blocking

Grooveek
  • 10,046
  • 1
  • 27
  • 37
0

Have a look at my answer in this thread

How to find out the optimal amount of threads?

Your example will likely be CPU bound, so you need a way to work out the contention to be able to work out the right number of threads on your box to use and be able to keep them all busy. Profiling will help there but remember it'll depend on the number of cores (as well as the network latency already mentioned etc) so use the runtime to get the number of cores when wiring up your thread pool size.

No quick answer I'm afraid, there will be an element of test, measure, adjust, repeat I'm afraid!

Community
  • 1
  • 1
Toby
  • 9,523
  • 8
  • 36
  • 59
-2

I say use something like Akka manage the threads for u. Use Jersey http client lib with non blocking IO which works with callback if i remember correctly. It's possibly the ideal setting for that type of tasks.

sanssucre
  • 17
  • 1