Java best way to determine the optimal number of threads

Question

I am writing a crawler which have a set number of worker threads that handle web IO and process each webpage. However, I am not sure how I can find the optimal number of threads for the system. What's the best way to monitor thread performance and concretely measure factors such as thrashing?

Answer 1

Just run benchmarks. You can do this dynamically by having threads report their throughput on a periodic basis to a thread manager. The manager can manipulate the thread count to probe how the performance varies with thread count and settle on a near-optimal value. It can also occasionally decide to experiment to see if a new optimal can be found. (The optimal count may vary with varying environmental conditions like overall CPU load from other processes.)