Why Sequential Semantic on x86/x86_64 is using through MOV [addr], reg + MFENCE instead of + SFENCE?

Question

At Intel x86/x86_64 systems have 3 types of memory barriers: lfence, sfence and mfence. The question in terms of their use. For Sequential Semantic (SC) is sufficient to use MOV [addr], reg + MFENCE for all memory cells requiring SC-semantics. However, you can write code in the whole and vice versa: MFENCE + MOV reg, [addr]. Apparently felt, that if the number of stores to memory is usually less than the loads from it, then the use of write-barrier in total cost less. And on this basis, that we must use sequential stores to memory, made ​​another optimization - [LOCK] XCHG, which is probably cheaper due to the fact that "MFENCE inside in XCHG" applies only to the cache line of memory used in XCHG (video where on 0:28:20 said that MFENCE more expensive that XCHG).

http://www.cl.cam.ac.uk/~pes20/cpp/cpp0xmappings.html

C/C++11 Operation x86 implementation

• Load Seq_Cst: MOV (from memory)
• Store Seq Cst: (LOCK) XCHG // alternative: MOV (into memory),MFENCE

Note: there is an alternative mapping of C/C++11 to x86, which instead of locking (or fencing) the Seq Cst store locks/fences the Seq Cst load:

• Load Seq_Cst: LOCK XADD(0) // alternative: MFENCE,MOV (from memory)
• Store Seq Cst: MOV (into memory)

The difference is that ARM and Power memory barriers interact exclusively with LLC (Last Level Cache), and x86 interact and with lower level caches L1/L2. In x86/x86_64:

• lfence on Core1: (CoreX-L1) -> (CoreX-L2) -> L3-> (Core1-L2) -> (Core1-L1)
• sfence on Core1: (Core1-L1) -> (Core1-L2) -> L3-> (CoreX-L2) -> (CoreX-L1)

In ARM:

• ldr; dmb;: L3-> (Core1-L2) -> (Core1-L1)
• dmb; str; dmb;: (Core1-L1) -> (Core1-L2) -> L3

C++11 code compiled by GCC 4.8.2 - GDB in x86_64:

std::atomic<int> a;
int temp = 0;
a.store(temp, std::memory_order_seq_cst);
0x4613e8  <+0x0058>         mov    0x38(%rsp),%eax
0x4613ec  <+0x005c>         mov    %eax,0x20(%rsp)
0x4613f0  <+0x0060>         mfence

But why on x86/x86_64 Sequential Semantic (SC) using through MOV [addr], reg + MFENCE, and not MOV [addr], reg + SFENCE, why do we need full-fence MFENCE instead of SFENCE there?

Answer 1

sfence doesn't block StoreLoad reordering. Unless there are any NT stores in flight, it's architecturally a no-op. Stores already wait for older stores to commit before they themselves commit to L1d and become globally visible, because x86 doesn't allow StoreStore reordering. (Except for NT stores / stores to WC memory)

For seq_cst you need a full barrier to flush the store buffer / make sure all older stores are globally visible before any later loads. See https://preshing.com/20120515/memory-reordering-caught-in-the-act/ for an example where failing to use mfence in practice leads to non-sequentially-consistent behaviour, i.e. memory reordering.

---

As you found, it is possible to map seq_cst to x86 asm with full barriers on every seq_cst load instead of on every seq_cst store / RMW. In that case you wouldn't need any barrier instructions on stores (so they'd have release semantics), but you'd need mfence before every atomic::load(seq_cst).

Answer 2

You don't need an mfence; sfence does indeed suffice. In fact, you never need lfence in x86 unless you are dealing with a device. But Intel (and I think AMD) has (or at least had) a single implementation shared with mfence and sfence (namely, flushing the store buffer), so there was no performance advantage to using the weaker sfence.

BTW, note that you don't have to flush after every write to a shared variable; you only have to flush between a write and a subsequent read of a different shared variable.