0

I create a shared memory used to share data between processes. The shared memory saves data using a data structure:

struct shared_cfg {
    volatile uint32_t idx;
    volatile uint32_t cfg_lock;
    ...
};

In my program:

  • different processes should get an unique value of "idx" every time.
  • after process P1 modified the viraible "idx", process P2 is expected to use the new value of "idx" and update it.
  • Then P1 will get the new value that saved by P2.
  • ...

But I found that when the concurrency increases, P2 occasionally read the old value of "idx", which means P1 and P2 got the same value of idx:

typedef volatile uint32_t lock_t

void lock_idx(lock_t *lock)
{
    uint32_t old_lock = 0;
    do {
        old_lock = __sync_val_compare_and_swap(lock, 0, 1);
        if (old_lock == 1)
            usleep(10*1000);
        else
            break;
    } while (old_lock == 1);
}
void unlock(lock_t *lock)
{
    __sync_lock_test_and_set(lock, 0);
}

idx = 0;

P1:

lock_idx(&cfg_lock);
idx = idx + 1;
unlock_idx(&cfg_lock);

P2

lock_idx(&cfg_lock);
idx = idx + 1;
unlock_idx(&cfg_lock);

I insert memory barrier by API "mm*fence" before and after the modification of "idx", but the problem still exist:

idx = 0;

P1:

lock_idx(&cfg_lock);
_mm_mfence()
idx = idx + 1
_mm_mfence()
unlock_idx(&cfg_lock);

P2:

lock_idx(&cfg_lock);
_mm_mfence()
idx = idx + 1
_mm_mfence()
unlock_idx(&cfg_lock);

The error log is like this:

P1: get lock 
P1: idx is 1; update idx to 2 
P1: unlock

P2: get lock 
P2: idx is 2; update idx to 3 
P2: unlock 

... 
P1: get lock 
P1: idx is 63; update idx to 64 
P1: unlock

P2: get lock 
P2: idx is 63; update idx to 64 
P2: unlock 
...

How to share variables between multiple processes by shared memory?

HnlyWk
  • 23
  • 5
  • Barriers don't create atomicity, neither does `volatile`. That's what `std::atomic` is for. (Or C11 `atomic_int`.) – Peter Cordes Apr 23 '23 at 09:33
  • The initial version of the question didn't mention anything about locking! Assuming that lock works, I notice your later critical sections code blocks don't include reading the old `idx` into a local variable. If you only read `idx` separately after an atomic increment, of course you'll sometimes get duplicates. That's what GCC `__atomic_fetch_add` (or the obsolete `__sync_fetch_and_add`) are for, or C11 `atomic_fetch_add` – Peter Cordes Apr 23 '23 at 09:39
  • `__sync` builtins are already full barriers, as strong as `_mm_mfence()`. Your lock looks correct, I think, if you're calling it correctly. It's weird to use `__sync_lock_test_and_set(&lock, 0)` to unlock, but it should work as well as `__sync_lock_release` which is just a release-store, unlike other `__sync` functions. i.e. it's like `atomic_store_explicit(&lock, 0, memory_order_release)`. – Peter Cordes Apr 23 '23 at 10:16
  • Anyway, this isn't a [mcve] of what you're doing. You aren't showing the code that captures the old and new values of `idx`, just critical sections that increment without capturing either old or new value. Like `fetch_add` without using the return value. – Peter Cordes Apr 23 '23 at 10:17
  • 1
    Thank you. I replaced "idx = idx + 1" with __async_fetch_and_add, and it works! – HnlyWk Apr 28 '23 at 08:32

0 Answers0