The problem here that I see is that you spin so many threads that you will overload the machine resource-wise by simply having to manage all the queued threads even if technically they don't try to run all at the same time. They will take RAM and in the absence of RAM, will use SWAP space - which will then bring down the machine in a blaze of non-glory.
I would use a queue (azure queue, msmq, Systems.Collections.Queue) to queue up all the objects, use a limited number of Threads that will process the file using Async methods described in your background link and then the thread is done executing check for the next item in the queue and process that one. My recommendation, is to use a non-memory queue - I will explain below. The main benefit is to save ram so that your software doesn't crash or slow down because the queue is too big.
Parallel.ForEach and such are great time savers but can really ruin the performance of your machine when you are dealing with a lot of items - and if the machine ever goes down then you cannot recover from it unless you have a checkpoint somewhere. Using a persistent queue will allow you to properly manage not just machine resources but also where you are in the process.
You can then scale this across multiple machines by using a persistent queue like MSMQ or if in the cloud, Azure queues. If you use a service that checks how big the azure queue is, you can even bring up instances from time to time to reduce the load and then terminate the extra instances.
This is the scenario that I would implement:
Use the standard ThreadPool size
When you detect a new file/batch - submit to the queue
Have an event fire every time you insert a new item in the queue (if memory queue)
Have a process check the queue (if persistent queue)
If a new item is in the queue, first check if you have space in the ThreadPool ignore if you don't (use a PEEK approach so you don't remove the item) - Add a worker to ThreadPool if there is space
The process thread (which runs under ThreadPool) should execute and then check if you have another item in the queue - if not - the thread dies, which is fine
Using this method, you could run this with 1 machine, or 50,000 - provided you use a persistent queue for more than 1 machine you won't have any problems. Of course, make sure you do a proper job of testing for duplicate items if you are using Azure Queues as you could be handed a queued item that's been given to another machine.
It's a simple approach, scalable and can if using a persistent queue (even a file system) recover from failure. It will not however overload the machine by abusing the resources by forcing it to manage a ThreadPool with 1 million+ items.
Hope this helps
