4

I have two processes: Process A is mapping large file (~170 GB - content constantly changes) into memory for writing with the flags MAP_NONBLOCK and MAP_SHARED:

MyDataType *myDataType; = (MyDataType*)mmap(NULL, sizeof(MyDataType), PROT_WRITE, MAP_NONBLOCK | MAP_SHARED , fileDescriptor, 0);

and every second I call msync:

msync((void *)myDataType, sizeof(MyDataType), MS_ASYNC);

This section works fine.

The problem occurs when process B is trying to read from the same file that process A is mapped to, process A does not respond for ~20 seconds. Process B is trying to read from the file something like 1000 times, using fread() and fseek(), small blocks (~4 bytes every time). Most of the content the process is reading are close to each other.

What is the cause for this problem? Is it related to pages allocation? How can I solve it?

BTW, same problem occur when I use mmap() in process B instead of simple fread().

ItayB
  • 10,377
  • 9
  • 50
  • 77

3 Answers3

4

msync() is likely the problem. It forces the system to write to disk, blocking the kernel in a write frenzy. In general on Linux (it's the same on Solaris BTW), it is a bad idea to use msync() too often. There is no need to call msync() for the synchronization of data between the memory map and the read()/write() I/O operations, this is a misconception that comes from obsolete HOWTOs. In reality, mmap() makes only the file system cache "visible" for a process. This means that the memory blocks the process changes are still under kernel control. Even if your process crashed, the changes would land on the disk eventually. Other processes would also still be serviced by the same buffer.

Here another answer on the subject mmap, msync and linux process termination The interesting part is the link to a discussion on realworldtech where Linus Torvalds himself explains how buffer cache and memory mapping work.

PS: fseek()/fread() pair is also probably better replaced by pread(). 1 system call is always better than 2. Also fseek()/fread() read always 4K and copies in a buffer, so if you have several small reads without fseek(), it will read from its local buffer and maybe miss updates in process A.

Community
  • 1
  • 1
Patrick Schlüter
  • 11,394
  • 1
  • 43
  • 48
  • Excellent answer. You could also add dat MAP_NONBLOCK doesn't make sense (here). – joop Dec 17 '15 at 14:37
  • 1
    I tried to remove the msync() line from the code and it's still happend. Thanks for the tip, I'll try to use pread() – ItayB Dec 17 '15 at 15:07
0

This sounds that you are suffering from IO-Starvation, which has nothing to do with the method (mmap or fread) you choose. You will have to improve your (pre-)caching-strategy and/or try another IO-scheduler (cfq being the default, maybe deadline delivers better overall-results for you)

You can change the scheduler by writing to /sys:

echo deadline > /sys/block/<device>/queue/scheduler
Ctx
  • 18,090
  • 24
  • 36
  • 51
  • It's indeed non-trivial to tackle this kind of problem; several measures (e.g. a raid-controller with bbu for writeback and faster disks) may be additionally necessary to improve the performance. – Ctx Dec 17 '15 at 15:34
0

Maybe you should try profiling or even using strace to figure out for sure where the process is spending its time. 20 s seems like an awfully long time to be explained by io in msync().

When you say A doesn't respond, what exactly do you mean?

Patrick Schlüter
  • 11,394
  • 1
  • 43
  • 48
Ziffusion
  • 8,779
  • 4
  • 29
  • 57
  • 1
    I have additional processes which sending keep alive messages to process A. When process B is trying to read from the file (even when I only call to fseek), Process B stops from answering to the keep alive, although I can see that he is still running (the main loop in the process keeps printing logs). – ItayB Dec 18 '15 at 07:30
  • In your OP you say "process A does not respond for ~20 seconds". Here you are indicating its process B. Seems like B is multithreaded (logs from the main loop). How is setup to respond to keep alives? Have you run strace on the process (whichever becomes unresponsive)? I would get a trace for the period it is hanging, and then for when it is hanging + not responding to keep alives. – Ziffusion Dec 18 '15 at 16:06