8

I am currently sharing data (<1KB) between multiple processes by memory-mapping. 1 "writer" process and multiple "reader" processes all mmap the same file.

Currently the reader processes need to constantly keep checking for updates. The reader processes keep polling the mmap-ed region to see if any new data is written.

Typical Usage (and existing implementation):
The "Writer" process is a logger which keeps appending new data (each on a new line) at irregular intervals. At any given point of time there can be one or more "reader" processes that are interested in any new data that the "writer" process generates. Also rather than having an indefinitely extending file, its is a circular buffer i.e. after a fixed number of lines the writer loops-back and start overwriting the file from the beginning with the new data. A header field in this file keeps track of position of the latest data i.e. the current "head".

In short the systems attempts to mimic the semantics of msgsnd() & msgrcv() with two additional caveats:

  1. Support multiple "readers"
    When "writer" posts a single msg, multiple notifications should be sent, 1 for each active "reader".
    --> Currently achieved as each "reader" constantly polls the "head" field and reads the new data when it changes.

  2. Persistence(file backed)
    If any "reader"/"writer" process is abruptly terminated, recovering the system should be as simple as restarting the process.
    --> Currently achieved as the IPC shared data is maintained in an mmap-ed file backed on the disk.

What would be a
- Fast (low latency) and
- Light-weight (low cpu-usage) alternative to implement some sort of event mechanism to notify the reader processes every time the mmap-ed region is modified by the writer process?

NOTE: In the reader process, adding an inotify watch on the mmap-ed file did NOT result in any events when the mmap-ed memory was updated by the writer process (even after calling msync()).

TheCodeArtist
  • 21,479
  • 4
  • 69
  • 130
  • From man page, `For file-backed mappings, the st_atime field for the mapped file may be updated at any time between the mmap() and the corresponding unmapping; the first reference to a mapped page will update the field if it has not been already. The st_ctime and st_mtime field for a file mapped with PROT_WRITE and MAP_SHARED will be updated after a write to the mapped region, and before a subsequent msync(2) with the MS_SYNC or MS_ASYNC flag, if one occurs.` - Can this be helpful ? – VoidPointer Jun 15 '13 at 11:33
  • 5
    You should use some kind of synchronization primitive (semaphore/mutex). You can put it in the mapped area. You should use it anyway to avoid race conditions, when one process writes and other one reads the same memory at the same time. – zch Jun 15 '13 at 11:37
  • @VoidPointer so in the reader process, i would have to constantly call stat() on the file repeatedly? – TheCodeArtist Jun 15 '13 at 12:03
  • @zch As there is always a single "writer", i believe we can do without any locking mechanism(to achieve faster performance). Hence I would like to avoid locking unless absolutely necessary. I have also updated the Q with the typical usage and details of the current implementation. – TheCodeArtist Jun 15 '13 at 12:22
  • @zch Basically, am trying to avoid using semaphores/mutexs so as NOT to reduce performance due to blocking (starvation) as explained in http://en.wikipedia.org/wiki/Readers-writers_problem. Since i just need an event i.e. notification mechanism to refactor the reader code into waiting for an event and not consume cpu cycles polling repeatedly. – TheCodeArtist Jun 15 '13 at 16:30
  • @TheCodeArtist , yes it needs polling using stat(). Other options, without thinking much on performance or consequences, did you thought of sending signals to readers after every write or using domain sockets. Check this for comparing mmap & domain sock : http://stackoverflow.com/questions/2101671/unix-domain-sockets-vs-shared-memory-mapped-file . – VoidPointer Jun 17 '13 at 05:16
  • TheCodeArtist: Only having a single writer is irrelevant. Simultaneous unsynchronized write+read by _any_ two threads is called a "data race", and it is undefined behavior in every threading model I know, including POSIX. Even if it happens to "work" on your system today, it can and will fail in hard-to-debug ways on your new system tomorrow. @zch is right: You must synchronize shared access, period. – Nemo Jun 20 '13 at 15:07
  • @Nemo Thank you for clarifying the point zch made. IIUC, when IPC using shared memory, is done without locking, the possibility of a "data race" makes the outcome of the IPC non-deterministic(order of execution depends on the scheduler/cpu-load and cannot be controlled). Any suggestions for a locking mechanism available to user-space programs(on Linux 2.6.37) in which the writer process never blocks/starves?... – TheCodeArtist Jun 21 '13 at 05:27
  • Just as a comment to the last part about inotify watches, the man pages lists that: The inotify API does not report file accesses and modifications that may occur because of mmap(2), msync(2), and munmap(2). – Nautilus Aug 17 '17 at 04:39

2 Answers2

2

(Posting the practically used solution for future reference)

Using this patch that adds mmap-ed file support to inotify,

  • reader processes can use the inotify framework
  • to monitor for changes on an mmap-ed file at significant/important intervals
  • that are triggerred by the writer process calling sync() on the same mmap-ed file.
TheCodeArtist
  • 21,479
  • 4
  • 69
  • 130
1

In all of the blocking communications, the reader grabs the data and no other reader can read it.

  • You can create a chain of readers. This is bad, because in your case, processes might end abruptly
  • Use multiple pipes. Each time a reader is created, it can ask the writer to open a pipe, and then the reader could subscribe to that pipe and read it. Also, the writer can send the reader a copy of the current state of data, unless readers can access it on its own.
  • Use signals. You could send a signal from the writer to all the readers after each write.

Tough question though... These are all I got...