Using pthread mutex shared between processes correctly

Question

There are many questions on stackoverflow about if a pthread mutex can be shared between processes, but I found no questions/answers regarding initialization of the shared mutex.

As far as I understand, the common way of using a process-shared mutex is the following: allocate a block of shared memory, initialize a pthread mutex on the shared memory block, use it.

In case of shared memory creation, it is handled by OS if multiple processes try to allocate a shared memory block with the same key ID. OK, but what I don't understand is how can I initialize a mutex on the shared memory block safely?

Am I right that the pthread_mutex_init doesn't provide any safe approach to initialize the pthread_mutex_t simultaneously from different processes? If yes, how can I provide exclusive access for processes to initialize a shared "mutual exclusion"? And how can I make sure if another process initialized the mutex successfully or not?

The second question relates to a case when a process blocking a mutex crashes. OK, there is a robust mutex which handles such cases and returns a corresponding error code. What about the shared memory block? It seems like a process should take care about if it is the last process which uses the shared memory to destroy it.

Answer 1

Am I right that the pthread_mutex_init doesn't provide any safe approach to initialize the pthread_mutex_t simultaneously from different processes?

Correct. It is up to you to ensure that only one process calls pthread_mutex_init() on the mutex, and that no process tries to operate on the mutex until that call has successfully returned.

For example, with POSIX shm_open() shared memory regions, you can have the processes try to open the region with the O_CREAT and O_EXCL flags, so that exactly one process will succeed in creating it. This process is then responsible for resizing the shared memory region and initialising the mutex with pthread_mutex_init(). The other processes must then wait for some kind of notification from the initialising process before opening the shared memory region - eg you could have the processes block opening a FIFO O_RDONLY, and have the initialising process notify them by opening the FIFO O_WRONLY (which will cause the open to succeed).

Usually, a shared memory segment will not be the only communication channel between the processes. Typically you would bootstrap the communication through a UNIX domain socket and negotiate the setup of the shared memory region over it, probably even passing the shared memory region file descriptor through the socket with a SCM_RIGHTS message. The shared memory region would then be used to accelerate the performance-sensitive IPC.

Answer 2

I use mmap to create a shared memory block.

int fd = open(filename, O_RDWR | O_CLOEXEC);

when the file does not exist, then fd < 0, then I try to initialize a mutex memory.

fd = open(filename, O_RDWR | O_CREAT | O_CLOEXEC | O_TRUNC | O_EXCL,
            S_IRUSR);
if(fd < 0) {
     // two processes might try to create the file simultaneously.
     // one open is success, the other one fail because of O_EXCL.
     sleep_random_10ms();
     continue; // continue and try again.
}

Note: because of S_IRUSR, after open is success and the file is created, the file is still not writable. Any other processes will not be able to open the file with O_RDWR, so that the first open still fail. No process will use the mutex while we are initializing the mutex.

after the file is created, we initialize a mutex as usual

pthread_mutexattr_init(&att);
pthread_mutexattr_setrobust(&att);
pthread_mutexattr_setpshared(&att, PTHREAD_PROCESS_SHARED);
pthread_mutex_init(&mutex, &att)
write(fd, &mutex, sizeof(mutex))
fchmod(fd, S_IRUSR | S_IWUSR)
close(fd);

The last two steps make the file writable so that any other process will open the file successfully at the first attempt.

We start to use the mutex as usual

mutex = (pthread_mutex_t*) mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED, fd_, 0);
pthread_mutex_lock(mutex);
....
pthread_mutex_unlock(mutex);

to clean up

munmap(&mutex);
close(fd);