2

I have a web app that allows you to paste a large number of urls in (anywhere from 2000 - 15000), it then visits each page to download the source and do some processing.

I've tried a couple methods, currently it uses ajax to send each url one at a time to a WebMethod which then uses ThreadPool.QueueUserWorkItem to push the request onto a queue and do the processing when it can so that many threads are running at the same time. This seems to cause problem (the Visual Studio web server dies occasionally which can't be good).

Just wondering if anyones done anything similar, knows a better way. I guess being able to throttle the number of concurrent threads somehow would help.

Thanks

JeremyBeadle
  • 683
  • 1
  • 8
  • 23
  • Cassini is not meant to handle this kind of load. You should use it for what it is for. – O.O Jan 13 '12 at 21:08

2 Answers2

2

Don't use the built in webserver. Use IIS proper

Jordan
  • 2,708
  • 4
  • 22
  • 35
  • I was going to change it (still will), but figured it's probably more to do with the amount of work it is doing in one go rather than the webserver? – JeremyBeadle Jan 13 '12 at 21:17
  • My guess is it is the webserver .... check this out http://www.developer.com/net/deciding-on-a-microsoft-web-server-cassini-iis-and-iis-express.html It is relatively easy to try setting it up with IIS7.5 Express and try it. Also what kind of processing is it doing? If you are doing multi-threading stuff perhaps you are running into deadlocks and such, not sure how much you know on that topic or precisely what you are doing so here's a link http://msdn.microsoft.com/en-us/magazine/cc163618.aspx – Jordan Jan 13 '12 at 22:33
1

Along with using IIS as mentioned above, you might benefit from using asynchronous web requests. (See How to use HttpWebRequest (.NET) asynchronously? for more info.)

This will prevent the amount of blocking done on each thread, which should reduce the number of threads needed if your processing isn't too heavy.

Community
  • 1
  • 1
Alex Moore
  • 3,415
  • 1
  • 23
  • 39