9

I need a copy-free re-size of a very large mmap file while still allowing concurrent access to reader threads.

The simple way is to use two MAP_SHARED mappings (grow the file, then create a second mapping that includes the grown region) in the same process over the same file and then unmap the old mapping once all readers that could access it are finished. However, I am curious if the scheme below could work, and if so, is there any advantage to it.

  1. mmap a file with MAP_PRIVATE
  2. do read-only access to this memory in multiple threads
  3. either acquire a mutex for the file, write to the memory (assume this is done in a way that the readers, which may be reading that memory, are not messed up by it)
  4. or acquire the mutex, but increase the size of the file and use mremap to move it to a new address (resize the mapping without copying or unnecessary file IO.)

The crazy part comes in at (4). If you move the memory the old addresses become invalid, and the readers, which are still reading it, may suddenly have an access violation. What if we modify the readers to trap this access violation and then restart the operation (i.e. don't re-read the bad address, re-calculate the address given the offset and the new base address from mremap.) Yes I know that's evil, but to my mind the readers can only successfully read the data at the old address or fail with an access violation and retry. If sufficient care is taken, that should be safe. Since re-sizing would not happen often, the readers would eventually succeed and not get stuck in a retry loop.

A problem could occur if that old address space is re-used while a reader still has a pointer to it. Then there will be no access violation, but the data will be incorrect and the program enters the unicorn and candy filled land of undefined behavior (wherein there is usually neither unicorns nor candy.)

But if you controlled allocations completely and could make certain that any allocations that happen during this period do not ever re-use that old address space, then this shouldn't be a problem and the behavior shouldn't be undefined.

Am I right? Could this work? Is there any advantage to this over using two MAP_SHARED mappings?

Eloff
  • 20,828
  • 17
  • 83
  • 112
  • You could just use a read/write lock and protect remapping under the write lock, no? – fge Jan 02 '12 at 18:01
  • I would guess that the bottleneck is the disk. Are you sure it is worth the trouble? copying huge files from disk always take time, because disks are slow mechanical devices. – Basile Starynkevitch Jan 02 '12 at 18:03
  • fge, yes but locking on the reading threads is out of the question in this case – Eloff Jan 02 '12 at 18:07
  • @BasileStarynkevitch Not necessarily. The file or at least the hot parts of it should fit entirely in memory, so re-sizing the mapping does not need to involve any file IO at all, other than the actual expansion of the file (and there are ways to mitigate that, e.g. increase the file size in a background thread well before running out of space.) – Eloff Jan 02 '12 at 18:11
  • I understood "very large file" as "file not fitting in RAM". – Basile Starynkevitch Jan 02 '12 at 18:12

1 Answers1

6

It is hard for me to imagine a case where you don't know the upper bound on how large the file can be. Assuming that's true, you could "reserve" the address space for the maximum size of the file by providing that size when the file is first mapped in with mmap(). Of course, any accesses beyond the actual size of the file will cause an access violation, but that's how you want it to work anyway -- you could argue that reserving the extra address space ensures the access violation rather than leaving that address range open to being used by other calls to things like mmap() or malloc().

Anyway, the point is with my solution, you never move the address range, you only change its size and now your locking is around the data structure that provides the current valid size to each thread.

My solution doesn't work if you have so many files that the maximum mapping for each file runs you out of address space, but this is the age of the 64-bit address space so hopefully your maximum mapping size is no problem.

(Just to make sure I wasn't forgetting something stupid, I did write a small program to convince myself creating the larger-than-file-size mapping gives an access violation when you try to access beyond the file size, and then works fine once you ftruncate() the file to be larger, all with the same address returned from the first mmap() call.)

andy
  • 6,888
  • 1
  • 19
  • 17
  • 1
    This was my first idea, but sadly address space is limited to 8TB in modern operating systems, which means that if there are a lot of memory mapped files all reserving the maximum possible space you can run out of address space. One could use mmap with MAP_NORESERVE to reserve as much address space as possible at program startup, and then allocate from there with MAP_FIXED, distributing it evenly at first and then taking from the maps that use it least when running out. Then use the above strategy or standard memcpy if a mmap runs out of space and there is no reserved adjacent space. – Eloff Jan 02 '12 at 19:25
  • where do you get your 8TB figure from ? – camelccc Nov 29 '12 at 16:05
  • 3
    Linux can manage 64TB of phsyiscal memory inside a 128TB address space. Not sure if you have a lot of files that are larger than 128TB (even the complete RAID volume on my network share is a mere 24TB, but your mileage may vary). You don't happen to be Google, do you? – Damon Jun 22 '14 at 14:23