Openmp: all threads stop due to delay in one thread

Question

I am new to OpenMP and Parallel programming and trying to experiment with it. I have a simple for loop of 30 elements and each of the elements are being processed by a process() function. But I deliberately delayed one element (the 5th element). Here's the code:

std::mutex mu;
void print_msg(const char* msg, size_t n)
{
    mu.lock();
    cout << msg << n << endl;
    mu.unlock();
}

void process(size_t i)
{
    print_msg("Processing ... ", i);

    if (i == 5) // the 5th element is big, so takes time
        for(int u=0; u<std::numeric_limits<int>::max(); u++);
}

int main(int argc, char *argv[])
{
    #pragma omp parallel
    {
        #pragma omp for ordered schedule(dynamic, 3)
        for(size_t i=0; i<30; i++)
        {
            process(i);
        }
    }

    return 0;
}

What I expected:

The thread that has been assigned the 5th element (and the other 2 elements) will get delayed but the rest will continue in parallel.

The result:

Here is the result with the delay position mentioned ...

Processing ... 1
Processing ... 0
Processing ... 4
Processing ... 2
Processing ... 5
Processing ... 3
Processing ... 6
Processing ... 7
Processing ... 8
[[ Here the execution paused for a couple of seconds and then
the below printed out all at once ]]
Processing ... 9
Processing ... 11
Processing ... 12
Processing ... 13
Processing ... 10
Processing ... 14
Processing ... 15
Processing ... 16
Processing ... 17
Processing ... 21
Processing ... 20
Processing ... 19
Processing ... 18
Processing ... 22
Processing ... 24
Processing ... 23
Processing ... 26
Processing ... 27
Processing ... 25
Processing ... 29
Processing ... 28

So it seems to me that not only the thread containing 5th element stopped, but all the other threads also stopped. Is it a normal behaviour ??

score 2 · Accepted Answer · answered Jul 26 '17 at 12:10

First, your thread is not "sleeping", but exercising the so-called "busy wait", which is not so great for long delays (see e.g. here: What are trade offs for “busy wait” vs “sleep”?).

But the real issue here seems to be the use of

#pragma omp for ordered schedule(dynamic, 3)

which basically means, that the threads will be executed in groups of 3 and the next group will wait for the result of the previous one (the particular groups execute sequentially). With dynamic the behavior is somewhat randomized, with static I'd expect you would see the pause after the groups 0, 1, 2 and 3, 4, 5 - with dynamic here it seems that the third group still made it to execute but the next group waits until the thread 5 finishes.

Try to remove the omp ordered line, that should then allow to execute all the threads in parallel and thus the other threads will not be blocked by execution of the thread 5.

Openmp: all threads stop due to delay in one thread

1 Answers1