52

A coworker and I write software for a variety of platforms running on x86, x64, Itanium, PowerPC, and other 10 year old server CPUs.

We just had a discussion about whether mutex functions such as pthread_mutex_lock() ... pthread_mutex_unlock() are sufficient by themselves, or whether the protected variable needs to be volatile.

int foo::bar()
{
 //...
 //code which may or may not access _protected.
 pthread_mutex_lock(m);
 int ret = _protected;
 pthread_mutex_unlock(m);
 return ret;
}

My concern is caching. Could the compiler place a copy of _protected on the stack or in a register, and use that stale value in the assignment? If not, what prevents that from happening? Are variations of this pattern vulnerable?

I presume that the compiler doesn't actually understand that pthread_mutex_lock() is a special function, so are we just protected by sequence points?

Thanks greatly.

Update: Alright, I can see a trend with answers explaining why volatile is bad. I respect those answers, but articles on that subject are easy to find online. What I can't find online, and the reason I'm asking this question, is how I'm protected without volatile. If the above code is correct, how is it invulnerable to caching issues?

starblue
  • 55,348
  • 14
  • 97
  • 151
David
  • 1,023
  • 1
  • 8
  • 16
  • 3
    possible duplicate of [Using C/Pthreads: do shared variables need to be volatile?](http://stackoverflow.com/questions/78172/using-c-pthreads-do-shared-variables-need-to-be-volatile) – Emile Cormier Jul 26 '11 at 23:16
  • 2
    See also: http://stackoverflow.com/questions/3208060/does-guarding-a-variable-with-a-pthread-mutex-guarantee-its-also-not-cached – Emile Cormier Jul 26 '11 at 23:21

7 Answers7

16

Simplest answer is volatile is not needed for multi-threading at all.

The long answer is that sequence points like critical sections are platform dependent as is whatever threading solution you're using so most of your thread safety is also platform dependent.

C++0x has a concept of threads and thread safety but the current standard does not and therefore volatile is sometimes misidentified as something to prevent reordering of operations and memory access for multi-threading programming when it was never intended and can't be reliably used that way.

The only thing volatile should be used for in C++ is to allow access to memory mapped devices, allow uses of variables between setjmp and longjmp, and to allow uses of sig_atomic_t variables in signal handlers. The keyword itself does not make a variable atomic.

Good news in C++0x we will have the STL construct std::atomic which can be used to guarantee atomic operations and thread safe constructs for variables. Until your compiler of choice supports it you may need to turn to the boost library or bust out some assembly code to create your own objects to provide atomic variables.

P.S. A lot of the confusion is caused by Java and .NET actually enforcing multi-threaded semantics with the keyword volatile C++ however follows suit with C where this is not the case.

AJG85
  • 15,849
  • 13
  • 42
  • 50
  • Thank you, I've read that several times online. However I really want to understand how I'm protected without volatile. – David Jul 26 '11 at 23:19
  • 2
    I didn't understand it myself until I read this: http://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf – AJG85 Jul 26 '11 at 23:28
15

Your threading library should include the apropriate CPU and compiler barriers on mutex lock and unlock. For GCC, a memory clobber on an asm statement acts as a compiler barrier.

Actually, there are two things that protect your code from (compiler) caching:

  • You are calling a non-pure external function (pthread_mutex_*()), which means that the compiler doesn't know that that function doesn't modify your global variables, so it has to reload them.
  • As I said, pthread_mutex_*() includes a compiler barrier, e.g: on glibc/x86 pthread_mutex_lock() ends up calling the macro lll_lock(), which has a memory clobber, forcing the compiler to reload variables.
ninjalj
  • 42,493
  • 9
  • 106
  • 148
  • Is your first point part of the C spec? Or is this compiler dependent? – Samuel Aug 30 '18 at 15:32
  • 1
    @Samuel: toolchain dependant. Before C11 (released in Dec 2011, IIRC), there was no notion of threads on the standard. – ninjalj Sep 01 '18 at 10:33
  • @Samuel "_Or is this compiler dependent?_" The point where calling a function w/ unknown body does unknown stuff? It's knowledge dependent. If you have mind reading ability you could try to know whether an external function needs access to a variable, otherwise, you just assume it does. – curiousguy Nov 05 '19 at 23:04
  • 1
    The usual mechanism is just to avoid providing inlineable definitions of `pthread_mutex_*`, so it's an opaque function call that the optimizer must assume reads and writes any/all globally-reachable memory. [How does a mutex lock and unlock functions prevents CPU reordering?](https://stackoverflow.com/q/50951011). (Exactly the same behaviour that `asm("" ::: "memory")` has.) So yes, the compiler can't keep shared-var values in registers across a call. (I dislike calling that "caching", since it leads to confusion with CPU caches and coherency, but this usage isn't technically wrong) – Peter Cordes Jul 14 '21 at 21:07
11

If the above code is correct, how is it invulnerable to caching issues?

Until C++0x, it is not. And it is not specified in C. So, it really depends on the compiler. In general, if the compiler does not guarantee that it will respect ordering constraints on memory accesses for functions or operations that involve multiple threads, you will not be able to write multithreaded safe code with that compiler. See Hans J Boehm's Threads Cannot be Implemented as a Library.

As for what abstractions your compiler should support for thread safe code, the wikipedia entry on Memory Barriers is a pretty good starting point.

(As for why people suggested volatile, some compilers treat volatile as a memory barrier for the compiler. It's definitely not standard.)

MSN
  • 53,214
  • 7
  • 75
  • 105
  • 1
    Interesting. I'm sure many platforms we support will die before they get C++0x, but it's something to look forward to in 5-10 years. I'll read that document, thanks. – David Jul 27 '11 at 00:07
  • That Boehm article is great - and a great blast from the past: "Multiprocessors are finally becoming mainstream." Wow - I do not miss 2004! – sage Mar 05 '17 at 20:00
  • 1
    Could you explain how a compiler would possibly not compile correctly the code presented in the Q? – curiousguy Nov 05 '19 at 22:34
  • 3
    *Until C++0x, it is not* This answer is incorrect. `pthread_mutex_lock()` and `pthread_mutex_unlock()` are [POSIX functions that guarantee memory access order is properly maintained](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_12): – Andrew Henle Nov 23 '19 at 03:08
  • 2
    (cont) "Applications shall ensure that access to any memory location by more than one thread of control (threads or processes) is restricted such that no thread of control can read or modify a memory location while another thread of control may be modifying it. Such access is restricted using functions that synchronize thread execution and also synchronize memory with respect to other threads. The following functions synchronize memory with respect to other threads: ... `pthread_mutex_lock()` ... `pthread_mutex_unlock()` ..." – Andrew Henle Nov 23 '19 at 03:09
  • 1
    @AndrewHenle: Indeed, the usual mechanism is for them to be non-inline (opaque) functions, thus the optimizer must assume they modify any globally-reachable variables like `_protected`. [How does a mutex lock and unlock functions prevents CPU reordering?](https://stackoverflow.com/q/50951011). For example they *don't* stop local variables like loop counters from staying in registers, unless their address has (potentially) escaped the function. For correctness, that has the same effect as the real behaviour: other threads may have modified any shared variable. – Peter Cordes Jul 14 '21 at 21:03
3

With the exception of the simplest spin lock algorithm, mutex code is quite involved: a good optimized mutex lock/unlock code contains the kind of code even excellent programmer struggle to understand. It uses special compare and set instructions, manages not only the unlocked/locked state but also the wait queue, optionally uses system calls to go into a wait state (for lock) or wake up other threads (for unlock).

There is no way the average compiler can decode and "understand" all that complex code (again, with the exception of the simple spin lock) no matter way, so even for a compiler not aware of what a mutex is, and how it relates to synchronization, there is no way in practice a compiler could optimize anything around such code.

That's if the code was "inline", or available for analyse for the purpose of cross module optimization, or if global optimization is available.

I presume that the compiler doesn't actually understand that pthread_mutex_lock() is a special function, so are we just protected by sequence points?

The compiler does not know what it does, so does not try to optimize around it.

How is it "special"? It's opaque and treated as such. It is not special among opaque functions.

There is no semantic difference with an arbitrary opaque function that can access any other object.

My concern is caching. Could the compiler place a copy of _protected on the stack or in a register, and use that stale value in the assignment?

Yes, in code that act on objects transparently and directly, by using the variable name or pointers in a way that the compiler can follow. Not in code that might use arbitrary pointers to indirectly use variables.

So yes between calls to opaque functions. Not across.

And also for variables which can only be used in the function, by name: for local variables that don't have either their address taken or a reference bound to them (such that the compiler cannot follow all further uses). These can indeed be "cached" across arbitrary calls include lock/unlock.

If not, what prevents that from happening? Are variations of this pattern vulnerable?

Opacity of the functions. Non inlining. Assembly code. System calls. Code complexity. Everything that make compilers bail out and think "that's complicated stuff just make calls to it".

The default position of a compiler is always the "let's execute stupidly I don't understand what is being done anyway" not "I will optimize that/let's rewrite the algorithm I know better". Most code is not optimized in complex non local way.

Now let's assume the absolute worse (from out point of view which is that the compiler should give up, that is the absolute best from the point of view of an optimizing algorithm):

  • the function is "inline" (= available for inlining) (or global optimization kicks in, or all functions are morally "inline");
  • no memory barrier is needed (as in a mono-processor time sharing system, and in a multi-processor strongly ordered system) in that synchronization primitive (lock or unlock) so it contains no such thing;
  • there is no special instruction (like compare and set) used (for example for a spin lock, the unlock operation is a simple write);
  • there is no system call to pause or wake threads (not needed in a spin lock);

then we might have a problem as the compiler could optimize around the function call. This is fixed trivially by inserting a compiler barrier such as an empty asm statement with a "clobber" for other accessible variables. That means that compiler just assumes that anything that might be accessible to a called function is "clobbered".

or whether the protected variable needs to be volatile.

You can make it volatile for the usual reason you make things volatile: to be certain to be able to access the variable in the debugger, to prevent a floating point variable from having the wrong datatype at runtime, etc.

Making it volatile would actually not even fix the issue described above as volatile is essentially a memory operation in the abstract machine that has the semantics of an I/O operation and as such is only ordered with respect to

  • real I/O like iostream
  • system calls
  • other volatile operations
  • asm memory clobbers (but then no memory side effect is reordered around those)
  • calls to external functions (as they might do one the above)

Volatile is not ordered with respect to non volatile memory side effects. That makes volatile practically useless (useless for practical uses) for writing thread safe code in even the most specific case where volatile would a priori help, the case where no memory fence is ever needed: when programming threading primitives on a time sharing system on a single CPU. (That may be one of the least understood aspects of either C or C++.)

So while volatile does prevent "caching", volatile doesn't even prevent compiler reordering of lock/unlock operation unless all shared variables are volatile.

curiousguy
  • 8,038
  • 2
  • 40
  • 58
3

The volatile keyword is a hint to the compiler that the variable might change outside of program logic, such as a memory-mapped hardware register that could change as part of an interrupt service routine. This prevents the compiler from assuming a cached value is always correct and would normally force a memory read to retrieve the value. This usage pre-dates threading by a couple decades or so. I've seen it used with variables manipulated by signals as well, but I'm not sure that usage was correct.

Variables guarded by mutexes are guaranteed to be correct when read or written by different threads. The threading API is required to ensure that such views of variables are consistent. This access is all part of your program logic and the volatile keyword is irrelevant here.

jbruni
  • 1,238
  • 8
  • 12
2

Locks/synchronisation primitives make sure the data is not cached in registers/cpu cache, that means data propagates to memory. If two threads are accessing/ modifying data with in locks, it is guaranteed that data is read from memory and written to memory. We don't need volatile in this use case.

But the case where you have code with double checks, compiler can optimise the code and remove redundant code, to prevent that we need volatile.

Example: see singleton pattern example
https://en.m.wikipedia.org/wiki/Singleton_pattern#Lazy_initialization

Why do some one write this kind of code? Ans: There is a performance benefit of not accuiring lock.

PS: This is my first post on stack overflow.

Obsidian
  • 3,719
  • 8
  • 17
  • 30
Gangadhar Mylapuram
  • 21
  • 1
  • CPU cache is coherent; locks don't need to flush it. Data being in CPU cache at all is sufficient for it to be globally visible. Don't confuse compilers choosing to hold variable values in registers with how hardware caches memory. Registers are per-core private, CPU-cache is coherent. See also https://software.rajivprab.com/2018/04/29/myths-programmers-believe-about-cpu-caches/ – Peter Cordes Jul 14 '21 at 21:08
  • You seem to be suggesting using `volatile` for lockless atomics. ([When to use volatile with multi threading?](https://stackoverflow.com/a/58535118)) That's a bad idea; C++11 `std::atomic` is a decade old; use it instead. (With `std::memory_order_relaxed` if you want it to compile to about the same asm you'd get with `volatile`, or `memory_order_acquire` is probably necessary for at least the 2nd step of a double-check. And BTW, compilers like GCC make asm that uses a double-check with a cheap read-only first check for function-local `static` variables with non-const initializers.) – Peter Cordes Jul 14 '21 at 21:11
  • The question is asking about C++. Volatile in Java is a different thing to volatile in C++. – ninjalj Jun 08 '23 at 14:18
0

Not if the object you're locking is volatile, eg: if the value it represents depends on something foreign to the program (hardware state). volatile should NOT be used to denote any kind of behavior that is the result of executing the program. If it's actually volatile what I personally would do is locking the value of the pointer/address, instead of the underlying object. eg:

volatile int i = 0;
// ... Later in a thread
// ... Code that may not access anything without a lock
std::uintptr_t ptr_to_lock = &i;
some_lock(ptr_to_lock);
// use i
release_some_lock(ptr_to_lock);

Please note that it only works if ALL the code ever using the object in a thread locks the same address. So be mindful of that when using threads with some variable that is part of an API.

viraltaco_
  • 814
  • 5
  • 14