4

Does std::condition_variable::notify_one() or std::condition_variable::notify_all() guarantee that non-atomic memory writes in the current thread prior to the call will be visible in notified threads?

Other threads do:

{
    std::unique_lock lock(mutex);
    cv.wait(lock, []() { return values[threadIndex] != 0; });
    // May a thread here see a zero value and therefore start to wait again?
}

Main thread does:

fillData(values); // All values are zero and all threads wait() before calling this.
cv.notify_all(); // Do need some memory fence or lock before this
                 // to ensure that new non-zero values will be visible
                 // in other threads immediately after waking up?

Doesn't notify_all() store some atomic value therefore enforcing memory ordering? I did not clarified it.

UPD: according to Superlokkus' answer and an answer here: we have to acquire a lock to ensure memory writes visibility in other threads (memory propagation), otherwise threads in my case may read zero values.

Also I missed this quote here about condition_variable, which specifically answers my question. Even an atomic variable has to be modified under a lock in a case when the modification must become visible immediately.

Even if the shared variable is atomic, it must be modified under the mutex in order to correctly publish the modification to the waiting thread.

Emil Kabirov
  • 559
  • 4
  • 14
  • Per [this](https://timsong-cpp.github.io/cppwp/thread.condition#general-3) I think we can say it is a synchronization point but I'm not 100% sure. – NathanOliver Jul 30 '21 at 13:09
  • If you don't use a lock before `cv.notify_all()`... How do you prevent that the thread you intend to wake-up isn't already running before? (e.g. by having received a spurious wake-up before) Please, don't get me wrong. The lock could/should be released before notifying but should guard the `fillData(values);`, shouldn't it? – Scheff's Cat Jul 30 '21 at 13:09
  • @Scheff'sCat Let's assume that all threads wait() and all values are zero at the moment when the main thread calls fillData() – Emil Kabirov Jul 30 '21 at 13:22
  • Please always post a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example). For example the definition of you values are missing and how the thread are started. You should still keep the snippets as duplication extra for clarfication. Then I also maybe would add a code based explanation. – Superlokkus Jul 30 '21 at 13:23
  • Note the added comment – Superlokkus Jul 30 '21 at 14:23

1 Answers1

3

I guess you are mixing up memory ordering of so called atomic values and the mechanisms of classic lock based synchronization.

When you have a datum which is shared between threads, lets say an int for example, one thread can not simply read it while the other thread might be write to it meanwhile. Otherwise we would have a data race.

To get around this for long time we used classic lock based synchronization: The threads share at least a mutex and the int. To read or to write any thread has to hold the lock first, meaning they wait on the mutex. Mutexes are build so that they are fine that this can happen concurrently. If a thread wins gettting the mutex it can change or read the int and then should unlock it, so others can read/write too. Using a conditional variable like you used is just to make the pattern "readers wait for a change of a value by a writer" more efficient, they get woken up by the cv instead of periodically waiting on the lock, reading, and unlocking, which would be called busy waiting.

So because you hold the lock in any after waiting on the mutex or in you case, correctly (mutex is still needed) waiting on the conditional variable, you can change the int. And readers will read the new value after the writer was able to wrote it, never the old. UPDATE: However one thing if have to add, which might also be the cause of confusion: Conditional variables are subject for so called spurious wakeups. Meaning even though you write did not have notified any thread, a read thread might still wake up, with the mutex locked. So you have to check if you writer actually waked you up, which is usually done by the writer by changing another datum just to notify this, or if its suitable by using the same datum you already wanted to share. The lambda parameter overload of std::condition_variable::wait was just made to make the checking and going back to sleep code looking a bit prettier. Based on your question now I don't know if you want to use you values for this job.

However at snippet for the "main" thread is incorrect or incomplete: You are not synchronizing on the mutex in order to change values. You have to hold the lock for that, but notifying can be done without the lock.

std::unique_lock lock(mutex);
fillData(values); 
lock.unlock();
cv.notify_all(); 

But these mutex based patters have some drawbacks and are slow, only one thread at a time can do something. This is were so called atomics, like std::atomic<int> came into play. They can be written and read at the same time without an mutex by multiple threads concurrently. Memory ordering is only a thing to consider there and an optimization for cases where you uses several of them in a meaningful way or you don't need the "after the write, I never see the old value" guarantee. However with it's default memory ordering memory_order_seq_cst you would also be fine.

Superlokkus
  • 4,731
  • 1
  • 25
  • 57
  • I am sorry, I am a little bit tired and also confused, so I want to sum up: do I HAVE to lock the mutex like you did in your snippet to ensure that after notify() other threads will see non-zero values? Yes or no :) We don't talk about data race when multiple threads may write to the same location, there is one writing thread. – Emil Kabirov Jul 30 '21 at 13:48
  • I suppose that without proper memory synchronization these values written by the main thread may remain in the main thread's CPU core cache and it may be not flushed to shared memory at the moment when other threads read the values – Emil Kabirov Jul 30 '21 at 13:52
  • 1
    You have to lock the mutex, as I said, in any case regardsless of writing or reading. Data races mean that, even if you have 5 as and old value and a write writes 7 without holding the lock, reader threads might read 5 or 7 or -455111785 or even worse. – Superlokkus Jul 30 '21 at 13:52
  • 1
    Your assumptions about computer architecture are A) not to be considered, since C++ is there to give you a portable way that works with every CPU and B) is faulty/incomplete since CPUs for long time have something as cache coherence protocols, so regardless of an actual write to the memory, your assumptions are usally with modern CPUs, wrong. So YES – Superlokkus Jul 30 '21 at 13:55
  • 1
    As I said YES to you first question. Plus what might confused you: You don't have to hold a lock if you read but NEVER write a datum, i.e. have 0 writers. For more than 0 writers, even only 1, you have to lock reading and writing. Or as I wrote, use an atomic. – Superlokkus Jul 30 '21 at 14:03
  • Thanks, and another question to clarify again: am I correct that even if there would be an atomic operation right after non-atomic writes in fillData(), I still have to lock the mutex to ensure that threads will read updated values? Because atomic ensures it's own value visibility and relevance, but it doesn't work like a memory barrier for non-atomic memory, right? – Emil Kabirov Jul 30 '21 at 14:05
  • 1
    As far as I understood you: Yes operations with atomic values are irrelevant for the regular ones, that whole memory order, "this value is ok if you read that one before" shebang is only for the sematics of atomic values like `std::atomic`. Don't forget to accept the answer ;-) – Superlokkus Jul 30 '21 at 14:08
  • After short thinking @EmilKabirov I have to add that if you mean by "memory barrier" that there is a so called "happens before" relation by the atomics, which would also imply what happend to the regular values, then No, i.e. you wouldn't have to lock a writer in general, but you would still have to in your case, since a reader, wakend by a spurious wake up, could be reading while you are writing. Or if you don't notifiy the readers only once forever in your case. – Superlokkus Jul 30 '21 at 14:19
  • In my case writer writes just 32 bit ints and readers read these ints, so atomicity of individual integer writes is guaranteed on modern machines. I was only currious about possibility when notify() wakes up another thread before a new written integer at the thread index becomes visible to the thread even if it was written prior to notify() – Emil Kabirov Jul 30 '21 at 14:29
  • Thats one of the many reasons, why I asked you about a minimal reproducible example. – Superlokkus Jul 30 '21 at 15:26