How to map shared memory to the same address in different processes?

Question

I'm trying to share some data between different processes but I would like to use the same address in all of them. From what I've read, it seems that this should be possible using mmap() with the MAP_FIXED flag (or MAP_FIXED_NOREPLACE). Some people suggest to create a shared block and then broadcast the address of that memory block to all the other processes. This comment is a good summary of the whole idea.

So I tried to implement that into this example. There are a sender and a receiver. The sender will create a shared block of memory. The receiver tries to allocate that block of memory at the same address. The address is passed around using a shared memory block too.

The issue is that the receiver can't map the memory object to said address. I get EINVAL error.

Is it possible to do what I'm trying? If so, how or what am I doing wrong in this example?

// header.h

#pragma once

#define SHM_DATA_ADDR "/bitarray-addr"
#define SHM_DATA "/bitarray-data"

#define NUM 3
#define SIZE (NUM * sizeof(int))
#define SIZE_ADDR (sizeof(void *))

// sender.c

#include "header.h"
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stddef.h>

int main() {
  // Open the shared memory object for read-only access
  // Set permissions
  int fd      = shm_open(SHM_DATA, O_CREAT | O_EXCL | O_RDWR, 0600);
  int fd_addr = shm_open(SHM_DATA_ADDR, O_CREAT | O_EXCL | O_RDWR, 0600);

  ftruncate(fd, SIZE);
  ftruncate(fd_addr, SIZE_ADDR);

  // Establish mapping between address space and memory object
  int RDWR = PROT_READ | PROT_WRITE;
  int *data        = (int *)mmap(0, SIZE, RDWR, MAP_SHARED, fd, 0);
  void **data_addr = (void **)mmap(0, SIZE_ADDR, RDWR, MAP_SHARED, fd_addr, 0);
  *data_addr = data;

  printf("%p\n", data);
  printf("%p\n", data_addr);
  printf("%p\n", *data_addr);

  // Unmap address space
  munmap(data_addr, SIZE);
  munmap(data, SIZE);

  // Close file descriptors
  close(fd_addr);
  close(fd);

  return EXIT_SUCCESS;
}

// receiver.c

#include "header.h"
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stddef.h>
#include <errno.h>

int main() {
  // Open the shared memory object for read-only access
  // Set permissions
  int fd      = shm_open(SHM_DATA, O_RDONLY, 0600);
  int fd_addr = shm_open(SHM_DATA_ADDR, O_RDONLY, 0600);

  ftruncate(fd, SIZE);
  ftruncate(fd_addr, SIZE_ADDR);

  // Establish mapping between address space and memory object
  void **data_addr = (void **)mmap(0, SIZE_ADDR, PROT_READ, MAP_SHARED, fd_addr, 0);
  int *data        = (int *)mmap(*data_addr, SIZE, PROT_READ, MAP_FIXED, fd, 0);

  printf("%p\n", data_addr);
  printf("%p\n", *data_addr);
  printf("%p\n", data); 

  // Unmap address space
  munmap(data_addr, SIZE_ADDR);
  munmap(data, SIZE);

  // Close file descriptors
  close(fd_addr);
  close(fd);

  // Destroy shared memory object
  shm_unlink(SHM_DATA_ADDR);
  shm_unlink(SHM_DATA);

  return EXIT_SUCCESS;
}

What possible reason could you have to want pointers in multiple processes to point to the same memory? You can just share an offset into the shared memory instead, making the entire exercise useless. — EOF, Nov 09 '20 at 21:11
It is quite obvious to me why this would be useful: you can share actual pointers into the mapped memory block between processes - a lot simpler for some applications than having to convert from pointer to page/offset back to pointer. — 500 - Internal Server Error, Nov 09 '20 at 21:21
@500-InternalServerError Why would that be simpler? C++ even has an abstraction of `std::pointer_traits` to make this as transparent as possible. Trying to force multiple processes to have a compatible memory layout is not a sane approach. — EOF, Nov 09 '20 at 21:39
@EOF, it's pretty much what @500 mentioned, I have structures with multiple layers of indirection. Structures that contain pointers to other structures that in turn point to other structures, etc... If each process has its own offset, I would need to shift all these intermediate pointers too, add an extra parameter to each function to pass around said offset, etc... Things would get messy. This might not be a perfect solution, but I think it could be a sensible one, before rewriting the whole codebase. Especially if there's meant to be a parameter in `mmap()` to do it. — orientnab, Nov 10 '20 at 00:13
@thatotherguy, I believe it's just the error thrown when map_fixed is used, but there is some issue with the address provided. The man page for mmap.3 says: _EINVAL The addr argument (if MAP_FIXED was specified) or off is not a multiple of the page size as returned by sysconf(), or is considered invalid by the implementation._ — orientnab, Nov 10 '20 at 00:19
@orientnab So what's the issue with the address provided? In my case, when I copy-pasted your code and ran it, `receiver` printed out `0x7ffff7ffb000` followed by `0x7ffff7ffb000` again. I.e. the first `mmap` returned the same address in `sender` and `receiver`, so since they mapped the files in opposite order, the address that `/bitarray-data` wanted was already in use by `/bitarray-addr` — that other guy, Nov 10 '20 at 00:48

score 3 · Accepted Answer · answered Nov 10 '20 at 02:38

From the mmap(2) man page, emphasis mine:

The flags argument determines whether updates to the mapping are visible to other processes mapping the same region, and whether updates are carried through to the underlying file. This behavior is determined by including exactly one of the following values in flags: [MAP_SHARED, MAP_SHARED_VALIDATE, MAP_PRIVATE]

In addition, zero or more of the following values can be ORed in flags: [MAP_ANONYMOUS, MAP_FIXED, etc]

In your receiver code, you have:

int *data        = (int *)mmap(*data_addr, SIZE, PROT_READ, MAP_FIXED, fd, 0);

which does not include any of MAP_SHARED, MAP_SHARED_VALIDATE, MAP_PRIVATE, hence the error. It's even stated explicitly in the Errors section:

EINVAL: flags contained none of MAP_PRIVATE, MAP_SHARED or MAP_SHARED_VALIDATE.

Change it to MAP_SHARED | MAP_FIXED and it runs successfully for me.

Hi, @Nate, brilliant, thanks! I completely missed that there had to be "exactly one" of those flags. Just changed it the way you suggested and it works perfectly. — orientnab, Nov 10 '20 at 07:27

How to map shared memory to the same address in different processes?

1 Answers1