Image processing of 100MP files takes too long, how to speed it up

Question

I'm working on an ASP.NET application that does a few imaging operations on large picture files (up to 100MP!). The processing of these pictures is actually relatively fast (less than a second) but the memory usage is understandably huge (about 500MB per image). When multiple pictures are uploaded to the server at the same time the server starts accepting all requests at the same time and the host runs out of memory.

1) How do I minimise this memory impact? 2) If the memory impact is minimised, there would still be a limit. So, can I also limit the absolute amount of images that are processed concurrently?

My own ideas and thoughts...

Because the execution time allows for some waiting (it's no problem if the requests takes a couple of seconds) I want to solve this by queuing the image transformation functions and only allowing concurrent execution of up to 2 or 3 pictures at the same time. This way, memory usage is at about 1.5GB which is fine. When moving to production, I'd like to increase this number as more memory will be available there.

Perhaps: How can I apply C# multi-threading classes (e.g. ConcurrentQueue, BlockingCollection, Interlocked) to ensure a single method invoked by a ASP.NET request handler can only execute in parallel a finite amount of instances?

Note that expensive threading operations are not really a problem here, as the overhead compared to the second-long operation of transforming the images is negligible.

public ActionResult UploadLargePicture()
{
    // Some trivial stuff like authorization

    var result = VeryMemoryIntensiveFunction(); // This is the part of the code that should have limited concurrency

    return Json(...);
}

Answer 1

1) Reduce memory demands on your image processing

Your ultimate problem is the inability to process concurrent requests because of memory limitations.

I suspect you are using .Net functions to manipulate the source image. Consider loading the image differently so it isn't cached in memory. The processing may (or may not) take a little longer, but it's much simpler and reliable than building queuing feature.

Instead of

Bitmap bmp = new Bitmap(pathToFile);

Use

var bmp = (Bitmap)Image.FromStream(sourceFileStream, false, false);

see https://stackoverflow.com/a/47424968/887092

There will still be a limit: Depending on how your system is being used, you still might need to limit the amount of concurrent requests.

2) The memory reductions should be enough, but here are some extra approaches to keeping a concurrent limit

Option A) Reject if the server is overloaded.

As each request arrives, keep track of how many current requests are already running. If it's higher than your defined limit then return an error in your RequestResponse. Perhaps you should use the HTTP Status code 429 - Too Many Requests in such a case.

try
{
    var currentImageCounter = Interlocked.Increment(ref imageCounter);
    if (currentImageCounter > 3)
    {
        throw new Exception(""); //Should be caught and result in HttpResponse.Status = 429
    }

    //Image processing code here

}
finally
{
    Interlocked.Decrement(ref ImageCounter);
}

Option A is best for response speed, but if overloaded, the user will get an error message.

Option B: Queue in Database

As each new image arrives, save it to disk, and add a record to the database, and trigger batch processing.

Have a batch process (console) that checks the table for incomplete work every X seconds OR upon trigger (can be a localhost HTTP request). Ensure the batch process only runs a single instance at a time (using a named Mutex/Semaphore).

Option B scales the most but doesn't respond quickly

Option C: Combination of A and B

Instead of rejecting when a threshold is reached (3), such ones should be queued in the database as a fallback.

Option D: A more detailed implementation of Option B that I wrote earlier

• A URL to upload workloads (ie. POST ./imageProcessing/Upload)
• Saving the images as files in App_Data with random filename (note: you'll need to add more features later to be sure orphaned files are deleted eventually), and any additional meta data in a relational database with the stored file name.
• Another URL which will process a single work item. (ie. GET ./imageProcessing/Process?id=737)
• A console application (which you can eventually make into a (Windows) Service), which calls the Process URL with parameter. The console application would work through the database table with work items, mark when processing starts and ends (StartAt, EndAt).
• The console application would could simply have separate threads to call the ProcessURL. The thread would get the next top 1 work item order by ID Primary Key. You could use a .Net Lock so that only a single thread can be fetching the next item at a time. You would be able to control how many threads are run at a time.
• Later, you could change from Threads to Async/Await. But that wouldn't really give you any noticeable performance benefit for what it's doing.
• The console application can be run manually on the desktop, and eventually packaged as a Windows Service.

Answer 2

I eventually went for a more integrated approach using some multi-threading constructs. Note that if it's not imperative for your project to immediately process the image (i.e. request is not waiting for the memory intensive function) @Todd answers is the way to go. The solution below works but can increase waiting times beyond the request timeout limit when a lot of images are uploaded concurrently.

using System;
using System.Threading.Tasks;
using System.Collections.Concurrent;
using System.Threading;

namespace MyApp.Services
{
    /// <summary>
    /// This service is meant to allow for scheduling memory intensive tasks,
    /// a maximum number of these types of tasks is defined, scheduling via
    /// this service guarantees no more than the max number of tasks are executed
    /// at once.
    /// </summary>
    public class MemoryIntensiveTaskService
    {
        private class ExecutionObject
        {
            public Func<object> Task { get; set; }
            public AutoResetEvent Event { get; set; }
            public object Result { get; set; }

            public T CastResult<T>()
            {
                return (T)Result;
            }
        }

        private static readonly int MaxConcurrency = 2;

        private static BlockingCollection<ExecutionObject> _queue = new BlockingCollection<ExecutionObject>();

        static MemoryIntensiveTaskService()
        {
            // Load MaxConcurrency number of consumers
            for (int i = 0; i < MaxConcurrency; i++)
                Task.Factory.StartNew(Consumer);
        }

        public T ScheduleTaskAndWait<T>(Func<T> action)
        {
            var executionObject = new ExecutionObject
            {
                Task = () => action(),
                Event = new AutoResetEvent(false),
                Result = null
            };

            // Add item to queue, will be picked up ASAP by a
            // consumer
            _queue.Add(executionObject);

            // Wait for completion
            executionObject.Event.WaitOne();

            return executionObject.CastResult<T>();
        }

        private static void Consumer()
        {
            while (true)
            {
                var executionObject = _queue.Take();

                // Execute task, store result
                executionObject.Result = executionObject.Task();

                // Fire event to signal to producer that execution
                // has finished
                executionObject.Event.Set();
            }
        }
    }
}