Threadpool WaitAll 64 [Leadtools]

Question

I am aware of the fact that there is a limit to WaitAll ManualResetEvents i.e. 64. (any reason why 64?) Due to this limitation, my below code does not work when SourceFolders.Count > 64.

if (Source.SubFolders.Count > 0)
{
    var threadPoolDoneEvents = new ManualResetEvent[Source.SubFolders.Count];
    for (int i = 0; i < Source.SubFolders.Count; i++)
    {
        try
        {
            if (m_bRegisterCancelled == false)
            {
                threadPoolDoneEvents[i] = new ManualResetEvent(false);
                AddSourceAgs variables = new AddSourceAgs();
                variables.Source = Source;
                variables.SubFolder = Source.SubFolders[i];
                variables.DoneEvents = threadPoolDoneEvents[i];
                ThreadPool.QueueUserWorkItem(AddSourceProcess, variables);
            }
            else
            {
                break;
            }
        }
        catch (System.Exception Ex)
        {
            FireError(this, "AddSource", this.ExBuilder.Message(ERROR_CASE_DATABASE_ADD_SOURCE_EX), Ex);
        }
    }
    WaitHandle.WaitAll(threadPoolDoneEvents);
}

I tried replacing the following line:

WaitHandle.WaitAll(threadPoolDoneEvents);

By this:

foreach (var waitHandle in threadPoolDoneEvents)
    waitHandle.WaitOne();

By this causes something unexpected to occur - in my case, out of 100 files, randomly 20 - 30 files are processed completely and then rest are stopped in between. Any idea where my concepts are falling behind? Any solution to this problem? I have tried this but this also causes some random errors like processing each task twice or something like that.

UPDATE After debugging, I found out that the tasks failed due to an exception thrown on this line of code:

try
{
    RasterImage image = m_codecs.Load(m_documentNameWithPath, pageNum);
    m_codecs.Save(image, m_outputPath+ "\\\\" + pageNum.ToString("D4") + ".png", RasterImageFormat.Png, 0);
    return true;
}
catch (Exception Ex)
{
    m_conversionErrors.Add(Ex);
    return false;
}

Following was the Exception thrown:

Function evaluation disabled because a previous function evaluation timed out. You must continue execution to reenable function evaluation.

Any solution do this? And even explain what this is?

This is a hard restriction in the underlying winapi function WaitForMultipleObjects(). Technically the limit could be increased to 128 but Microsoft refuses to fix it. Surely because they don't think programmers should write programs that use more than 64 threads. Which is about accurate for your code, it will only perform well with *one* thread. You only have one disk, it does *not* like to be commandeered around by multiple threads. The disk seeks can completely kill the perf of your app. The file system cache will hide this problem when you debug, that won't happen in production. — Hans Passant, Oct 16 '14 at 12:55

score 1 · Accepted Answer · answered Oct 16 '14 at 12:33

1

You waiting for those events does not cause the tasks to fail. More likely there is an exception which you do not notice (uncaught or swallowed).

Make your processing Task based and use Task.WaitAll. No limits there. You'll also get automatic error propagation.

answered Oct 16 '14 at 12:33

usr

168,620
35
240
369

Yes there is an exception that gets thrown because of which the tasks do not complete successfully. I have updated my question about the details of that exception. – amyn Oct 17 '14 at 05:17
This is just a debugger message. In any case this is immaterial to the question. There's a bug in the image conversion code. Has nothing to do with tasks or wait handles. – usr Oct 17 '14 at 08:27
Yes I was able to identify the error and the problem lies the section `m_codecs.load()`. But the problem was resolved after I replaced `ThreadPool.QueueUserWorkItem()` with `Parallel.For`. Changed nothing else in the `m_codecs.load()` section. – amyn Oct 20 '14 at 05:28

Threadpool WaitAll 64 [Leadtools]

1 Answers1