More appropriate for my task: background worker or thread pool?

Question

I have a simple web application module which basically accepts requests to save a zip file on PageLoad from a mobile client app.

Now, What I want to do is to unzip the file and read the file inside it and process it further..including making entries into a database.

Update: the zip file and its contents will be fairly smaller in size so the server shouldn't be burdened with much load.

Update 2: I just read about when IIS queues requests (at global/app level). So does that mean that I don't need to implement complex request handling mechanism and the IIS can take care of the app by itself?

Update 3: I am looking for offloading the processing of the downloaded zip not only for the sake of minimizing the overhead (in terms of performance) but also in order to avoid the problem of table-locking when the file is processed and records updated into the same table. In the scenario of multiple devices requesting the page and the background task processing database updateing in parallel would cause an exception.

As of now I have zeroed on two solutions:

• To implement a concurrent/message queue
• To implement the file processing code into a separate tool and schedule a job on the server to check for non-processed file(s) and process them serially.

Inclined towards a Queuing Mechanism I will try to implement is as it seems less dependent on config. v/s manually configuring the job/schedule at the server side.

So, what do you guys recommend me for this purpose?

Moreover after the zip file is requested and saved on server side, the client & server side connection is released after doing so. Not looking to burden my IIS.

Imagine a couple of hundred clients simultaneously requesting the page..

I actually haven't used neither of them before so any samples or how-to's will be more appreciated.

Answer 1

I'd recommend TPL and Rx Extensions: you make your unzipped file list an observable collection and for each item start a new task asynchronously.

Answer 2

I'd suggest a queue system.

When you received a file you'll save the path into a thread-synchronized queue. Meanwhile a background worker (or preferably another machine) will check this queue for new files and dequeue the entry to handle it.

This way you won't launch an unknown amount of threads (every zip file) and can handle the zip files in one location. This way you can also easier move your zip-handling code to another machine when the load gets too heavy. You just need to access a common queue.

The easiest would probably be to use a static Queue with a lock-object. It is the easiest to implement and does not require external resources. But this will result in the queue being lost when your application recycles.

You mentioned losing zip files was not an option, then this approach is not the best if you don't want to rely on external resources. Depending on your load it may be worth to utilize external resources - meaning upload the zip file to a common storage on another machine and add a message to an queue on another machine.

Here's an example with a local queue:

ConcurrentQueue<string> queue = new ConcurrentQueue<string>();

void GotNewZip(string pathToZip)
{
    queue.Enqueue(pathToZip); // Added a new work item to the queue
}

void MethodCalledByWorker()
{
    while (true)
    {
        if (queue.IsEmpty)
        {
            // Supposedly no work to be done, wait a few seconds and check again (new iteration)
            Thread.Sleep(TimeSpan.FromSeconds(5));
            continue;
        }

        string pathToZip;
        if (queue.TryDequeue(out pathToZip)) // If TryDeqeue returns false, another thread dequeue the last element already
        {
            HandleZipFile(pathToZip);
        }
    }
}

This is a very rough example. Whenever a zip arrives, you add the path to the queue. Meanwhile a background worker (or multiple, the example s threadsafe) will handle one zip after another, getting the paths from the queue. The zip files will be handled in the order they arrive.

You need to make sure that your application does not recycle meanwhile. But that's the case with all resources you have on the local machine, they'll be lost when your machine crashes.

Answer 3

I believe you are optimising prematurely.

You mentioned table-locking - what kind of db are you using? If you add new rows or update existing ones most modern databases in most configurations will:

1. use row-level locking; and
2. be fast enough without you needing to worry about locking.

I suggest starting with a simple method

        //Unzip
        //Do work
        //Save results to database

and get some proof it's too slow.