System-wide global variable / semaphore / mutex in C++/Linux?

Question

Is it possible to create a system-wide global variable / semaphore / mutex in C++ on Linux?

Here's the reason: I've got a system that often runs multiple copies of the same software on unrelated data. It's common to have 4 jobs, each running the same software. The software has a small section where it creates a huge graph that takes a lot of memory; outside that section memory usage is moderate.

It so happens sometimes that 2 jobs simultaneously hit the same memory-hungry section and the whole system starts swapping. Thus we want to prevent that by creating something like a critical section mutex between different jobs so that no more than one of them would allocate a lot of memory at a time.

If these were thread of the same job pthread locks would do the job.

What would be a good way to implement such mutex between different jobs?

Answer 1

You can use a named semaphore if you can get all the processes to agree on a common name.

A named semaphore is identified by a name of the form /somename; that is, a null-terminated string of up to NAME_MAX-4 (i.e., 251) characters consisting of an initial slash, followed by one or more characters, none of which are slashes. Two processes can operate on the same named semaphore by passing the same name to sem_open(3).

Answer 2

For interprocess mutual exclusion, you can use file locking. With linux, the code is as simple as protecting the critical section with a call to flock.

int fd_lock = open(LOCK_FILE, O_CREAT);

flock(fd_lock, LOCK_EX);

// do stuff

flock(fd_lock, LOCK_UN);

If you need POSIX compatibility, you can use fcntl.

Answer 3

You can make C++ mutexes work across process boundaries on Linux. However, there's some black magic involved which makes it less appropriate for production code.

Explanation:

The standard library's std::mutex and std::shared_mutex use pthread's struct pthread_mutex_s and pthread_rwlock_t under the hood. The native_handle() method returns a pointer to one of these structures.

The drawback is that certain details are abstracted out of the standard library and defaulted in the implementation. For example, std::shared_mutex creates its underlying pthread_rwlock_t structure by passing NULL as the second parameter to pthread_rwlock_init(). This is supposed to be a pointer to a pthread_rwlockattr_t structure containing an attribute which determines sharing policy.

public:
    __shared_mutex_pthread()
    {
        int __ret = pthread_rwlock_init(&_M_rwlock, NULL);
        ...

In theory, it should receive default attributes. According to the man pages for pthread_rwlockattr_getpshared():

The default value of the process-shared attribute is PTHREAD_PROCESS_PRIVATE.

That said, both std::shared_mutex and std::mutex work across processes anyway. I'm using Clang 6.0.1 (x86_64-unknown-linux-gnu / POSIX thread model). Here's a description of what I did to check:

• Create a shared memory region with shm_open.

• Check the size of the region with fstat to determine ownership. If .st_size is zero, then ftruncate() it and the caller knows that it is the region's creating process.

• Call mmap on it.

  • The creator process uses placement-new to construct a std::mutex or std::shared_mutex object within the shared region.
  • Later processes use reinterpret_cast<>() to obtain a typed pointer to the same object.

• The processes now loop on calling trylock() and unlock() at intervals. You can see them blocking one another using printf() before and after trylock() and before unlock().

Extra detail: I was interested in whether the c++ headers or the pthreads implementation were at fault, so I dug into pthread_rwlock_arch_t. You'll find a __shared attribute which is zero and a __flags attribute which is also zero for the field denoted by __PTHREAD_RWLOCK_INT_FLAGS_SHARED. So it seems that by default this structure is not intended to be shared, though it seems to provide this facility anyway (as of July 2019).

Summary

It seems to work, though somewhat by chance. I would advise caution in writing production software that works contrary to documentation.

Answer 4

I looked at using the shared-pthread-mutex solution but didn't like the logic race in it. So I wrote a class to do this using the atomic builtins

#include <string>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>

using std::string;

//from the command line - "ls /dev/shm" and "lsof /dev/shm/<name>" to see which process ID has access to it

template<typename PAYLOAD>
class InterprocessSharedVariable
{
protected:
    int mSharedMemHandle;
    string const mSharedMemoryName;
    bool mOpenedMemory;
    bool mHaveLock;
    pid_t mPID;

    // this is the shared memory structure
    typedef struct 
    {
        pid_t mutex;
        PAYLOAD payload;
    }
    tsSharedPayload;


    tsSharedPayload* mSharedData;


    bool openSharedMem()
    {
        mPID = getpid();

        // The following caters for the shared mem being created by root but opened by non-root,
        //  giving the shared-memory 777 permissions.
        int openFlags = O_CREAT | O_RDWR;
        int shareMode = S_IRWXU | S_IRWXG | S_IRWXO;

        // see https://stackoverflow.com/questions/11909505/posix-shared-memory-and-semaphores-permissions-set-incorrectly-by-open-calls
        // store old
        mode_t old_umask = umask(0);

        mSharedMemHandle = shm_open (mSharedMemoryName.c_str(), openFlags, shareMode);

        // restore old
        umask(old_umask);

        if (mSharedMemHandle < 0) 
        {
            std::cerr << "failed to open shared memory"  << std::endl;
            return false;
        }

        if (-1 == ftruncate(mSharedMemHandle, sizeof(tsSharedPayload)))
        {
            std::cerr <<  "failed to resize shared memory" << std::endl;
            return false;
        }

        mSharedData = (tsSharedPayload*) mmap (NULL, 
                                            sizeof(tsSharedPayload),
                                            PROT_READ | PROT_WRITE,
                                            MAP_SHARED,
                                            mSharedMemHandle,
                                            0);

        if (MAP_FAILED == mSharedData)
        {
            std::cerr << "failed to map shared memory" << std::endl;
            return false;
        }

        return true;
    }


    void closeSharedMem()
    {
        if (mSharedMemHandle > 0)
        {
            mSharedMemHandle = 0;
            shm_unlink (mSharedMemoryName.c_str());
        }
    }

public:
    InterprocessSharedVariable () = delete;

    InterprocessSharedVariable (string const&& sharedMemoryName) : mSharedMemoryName(sharedMemoryName)
    {
        mSharedMemHandle = 0;
        mOpenedMemory = false;
        mHaveLock = false;
        mPID = 0;
    }

    virtual ~InterprocessSharedVariable ()
    {
        releaseSharedVariable ();
        closeSharedMem ();
    }

    // no copying
    InterprocessSharedVariable (InterprocessSharedVariable const&) = delete;
    InterprocessSharedVariable& operator= (InterprocessSharedVariable const&) = delete;


    bool tryLockSharedVariable (pid_t& ownerProcessID)
    {
        // Double-checked locking.  See if a process has already grabbed the mutex.  Note the process could be dead
        __atomic_load (&mSharedData->mutex, &ownerProcessID, __ATOMIC_SEQ_CST);

        if (0 != ownerProcessID)
        {
            // It is possible that we have started with the same PID as a previous process that terminated abnormally
            if (ownerProcessID == mPID)
            {
                // ... in which case, we already "have ownership"
                return (true);
            }

            // Another process may have the mutex.  Check whether it is alive.
            // We are specifically looking for an error returned with ESRCH
            // Note that if the other process is owned by root, "kill 0" may return a permissions error (which indicates the process is running!)
            int processCheckResult = kill (ownerProcessID, 0);

            if ((0 == processCheckResult) || (ESRCH != errno))
            {
                // another process owns the shared memory and is running
                return (false);
            }

            // Here: The other process does not exist ((0 != processCheckResult) && (ESRCH == errno))
            // We could assume here that we can now take ownership, but be proper and fall into the compare-exchange
            ownerProcessID = 0;
        }

        // It's possible that another process has snuck in here and taken ownership of the shared memory.
        // If that has happened, the exchange will "fail" (and the existing PID is stored in ownerProcessID)

        // ownerProcessID == 0 -> representing the "expected" value
        mHaveLock = __atomic_compare_exchange_n (&mSharedData->mutex,
                                                &ownerProcessID,      //"expected"
                                                mPID,                 //"desired"
                                                false,                //"weak"
                                                __ATOMIC_SEQ_CST,     //"success-memorder"
                                                __ATOMIC_SEQ_CST);    //"fail-memorder"

        return (mHaveLock);
    }


    bool acquireSharedVariable (bool& failed, pid_t& ownerProcessID)
    {
        if (!mOpenedMemory)
        {
            mOpenedMemory = openSharedMem ();

            if (!mOpenedMemory)
            {
                ownerProcessID = 0;
                failed = true;
                return false;
            }
        }

        // infrastructure is working
        failed = false;

        bool gotLock = tryLockSharedVariable (ownerProcessID);
        return (gotLock);
    }

    void releaseSharedVariable ()
    {
        if (mHaveLock)
        {
            __atomic_store_n (&mSharedData->mutex, 0, __ATOMIC_SEQ_CST);
            mHaveLock = false;
        }
    }
};

Example usage - here we are simply using it to ensure that only one instance of the application runs.

int main(int argc, char *argv[])
{
    typedef struct { } tsEmpty;
    InterprocessSharedVariable<tsEmpty> programMutex ("/run-once");

    bool memOpenFailed;
    pid_t ownerProcessID;
    if (!programMutex.acquireSharedVariable (memOpenFailed, ownerProcessID))
    {
        if (memOpenFailed)
        {
            std::cerr << "Failed to open shared memory" << std::endl;
        }
        else
        {
            std::cerr << "Program already running - process ID " << ownerProcessID << std::endl;
        }
        return -1;
    }

    ... do stuff ...

    return 0;
}

Answer 5

Mutual exclusion locks (mutexes) prevent multiple threads from simultaneously executing critical sections of code that access shared data (that is, mutexes are used to serialize the execution of threads). All mutexes must be global. A successful call for a mutex lock by way of mutex_lock() will cause another thread that is also trying to lock the same mutex to block until the owner thread unlocks it by way of mutex_unlock(). Threads within the same process or within other processes can share mutexes.

Mutexes can synchronize threads within the same process or in other processes. Mutexes can be used to synchronize threads between processes if the mutexes are allocated in writable memory and shared among the cooperating processes (see mmap(2)), and have been initialized for this task.

For inter-process synchronization, a mutex needs to be allocated in memory shared between these processes. Since the memory for such a mutex must be allocated dynamically, the mutex needs to be explicitly initialized using mutex_init(). also, for inter-process synchronization, besides the requirement to be allocated in shared memory, the mutexes must also use the attribute PTHREAD_PROCESS_SHARED, otherwise accessing the mutex from another process than its creator results in undefined behaviour (see this: linux.die.net/man/3/pthread_mutexattr_setpshared): "The process-shared attribute is set to PTHREAD_PROCESS_SHARED to permit a mutex to be operated upon by any thread that has access to the memory where the mutex is allocated, even if the mutex is allocated in memory that is shared by multiple processes."