Problem with pThread sync issue

Question

I am facing a sync issue with pthread. threadWaitFunction1, is a thread wait function. I expect line no. 247 flag = 1 to be executed only after 243-246 has finished. But i find it strange that sometimes, it jumps directly to 247 before 243-246 has finished.

Please help me.

Thanks in advance.

236   struct timespec timeToWait;
237   static void* threadWaitFunction1(void *timeToWaitPtr)
238   {
239       cout << "Setting flag =0 inside threadWaitFunction1\n";
240       
241       cout << "Inside threadWaitFunction\n";
242       struct timespec *ptr = (struct timespec*) timeToWaitPtr;
243       pthread_mutex_lock(&timerMutex);
          flag = 0;
244       pthread_cond_timedwait(&timerCond, &timerMutex, ptr);
          flag=1;
245       pthread_mutex_unlock(&timerMutex);
246       cout << "Setting flag =1 inside threadWaitFunction1\n";
247       
248
249    }

The thread which creates and calls the above thread is:

263  static void timer_trackStartTime ()
264  {
265       struct timeval now;
266       pthread_t thread;
267       
268       printf("Inside trackStartTime: flag = %d\n",flag);
269 
270      /* Setting timer expiration */
271       timeToWait.tv_sec = lt_leak_start_sec;;  // First expiry after 1 sec
272       timeToWait.tv_nsec = lt_leak_start_nsec;
273       pthread_create(&thread, NULL, threadWaitFunction1, &timeToWait);
274       pthread_join(thread, NULL);
275       //pthread_kill(thread, SIGKILL); // Destroying the thread to ensure no leaks
276 
.
.
283       }

If i protect the whole function using pthread_mutex_lock, but still the same problem persists. How to ensure orderly execution? Can anyone help?

EDIT: now.tv_sec and now.tv_nsec removed from the code. *EDIT: Changed the flags inside the mutex (still does not work)*

Answer 1

So it is not really execution ordering (which is most probably correct) but timing that makes you unhappy. And under "it jumps directly to 247 before 243-246 has finished" you mean "I observed it executing 247 before the time it should wait in 244 has passed". Right?

Then, I suspect this is the problem of spurious wakeup: a thread might get woken up even though no other thread signalled the condition variable. The specification of pthread_cond_timedwait() says that "Spurious wakeups from the pthread_cond_timedwait() or pthread_cond_wait() functions may occur."

Usually, a condition variable is associated with a certain event in the application, and a thread waiting on a condition variable in fact waits for a signal by another thread that the event of interest has happened. If you have no event and just want to wait for a certain amount of time, indeed other ways, such as usleep() or timers, are more appropriate, except if you also need a pthread cancellation point.

ADDED: Since you seem satisfied with usleep() and only asked why pthread_cond_timedwait() did not work to your expectations, I decided not to post the code. If you need it, you may use the answer of @Hasturkun.

---

ADDED-2: The output in comments below (obtained after the solution of Hasturkun was applied) suggests that the waiting thread does not exit the loop, which likely means that pthread_cond_timedwait() returns something different than ETIMEDOUT. Have you seen the comment by @nos to your post (I fixed the amount of nanosecs to subtract):

Make sure (now.tv_usec * 1000) + lt_leak_start_nsec; doesn't overflow. You can only set tv_nsec to max 999999999, if the expression is larger than that, you should subtract 1000000000 from tv_nsec, and increment tv_sec by 1. If your timeToWaitPtr contains an invalid tv_nsec (larger than 999999999), pthread_cond_timedwait will fail (you should check its return value too.) – nos Apr 28 at 19:04

In this case, pthread_cond_timedwait() will repeatedly return EINVAL and will never get out of the loop. It is better to adjust the timeout before entering the wait loop, though it can also be done in response to EINVAL.

---

ADDED-3: Now after you changed the code in your question to pass the timeout without adding to current time, it has another problem. As stated in the spec, the timeout for pthread_cond_timedwait() is absolute time, not relative; so when you pass something like 3 sec as the timeout, it is interpreted as "3 seconds since the reference point for the system time". That moment is almost certainly passed for a while, and so pthread_cond_timedwait() returns immediately.
I would recommend you to read the specification thoroughly (including rationale) to build better understanding of how this function is supposed to be used.

Answer 2

Paul E. McKenney has written a book titled "Is Parallel Programming Hard, And, If So, What Can You Do About It?", which has really good information (and some nice pictures) on memory barriers.

Back to your question, flag isn't protected by anything. While you may think that pthread_mutex_lock() and pthread_mutex_unlock provides some strong ordering and visibility guarantees, the only guarantees it provides are for accesses inside the critical region and for the mutex itself.

What's more, on some architectures pthread_mutex_lock() uses an acquire barrier and pthread_mutex_unlock() uses a release barrier, which means that accesses before and after the mutex protected region can spill into the mutex protected region. Strong ordering guarantees are provided between a CPU releasing a mutex and another CPU acquiring the same mutex, but pretty much everything else doesn't need (and maybe doesn't get) such strong guarantees.

Edit:

Apparently I was wrong with respect to pthreads, they seem to require full memory barriers (if you interpret synchronize memory with respect to other threads as requiring that). More about this, and some info on the guarantees provided in real-world implementations at Reordering Constraints for Pthread-Style Locks by Hans Boehm.

I'm also still wondering about NPTL on IA64 1, 2.

Answer 3

As stated by Alexey Kukanov, the problem is likely spurious wakeup. your code may be corrected to loop until timeout occurs. Note that I also moved the flag setting to be under the mutex.

static void* threadWaitFunction1(void *timeToWaitPtr)
{
    struct timespec *ptr = (struct timespec*) timeToWaitPtr;
    int ret;

    pthread_mutex_lock(&timerMutex);
    cout << "Setting flag =0 inside threadWaitFunction1\n";
    flag=0;
    cout << "Inside threadWaitFunction\n";
    while (pthread_cond_timedwait(&timerCond, &timerMutex, ptr) != ETIMEDOUT)
        ;
    cout << "Setting flag =1 inside threadWaitFunction1\n";
    flag=1;
    pthread_mutex_unlock(&timerMutex);
}

To be on the safe side, you should check the flag under the same mutex to establish ordering

Answer 4

This could be because the compiler has optimized things and put your assignment to your flag before the thread mutex. If you want to guarantee order of execution, (something which is not normally guaranteed, on the only condition that the visible behaviour of your program does not change due to optimizations), you use a memory barrier to make sure that the instructions you want to be executed in the order you write them, are executed only in that order.

Here is a very interesting, though rather technical and long, article on how memory barriers work and what they do and don't do. It's written for Linux, but the basic principles remain the same.

EDIT:

The lock is an implicit memory barrier, by the link I gave earlier, so no memory barrier is needed.

Answer 5

Just for everyone's info:

What i could not achieve using pthread_cond_timedwait(&timerCond, &timerMutex, ptr); i have achieved using usleep( ), usleep takes timespec structure where we can specify the wait period using seconds and nanoseconds, and my purpose is solved.

So what does the pthread_cond_timedwait(&timerCond, &timerMutex, ptr); make sense for?? I am surprised, as this API is expected to make the calling thread wait, fo that condition to satisfy, but it seems that processor jumps to next instruction as an optimisation measure, and does not wait foer the condition to satisfy.

But still my problem remains the same, as to why, pthread_cond_timedwait(&timerCond, &timerMutex, ptr); should not make the calling thread wait?

It seems i wasted a day behind this API: pthread_cond_timedwait( )