6

In Python on Windows I can create a large file by

    from mmap import mmap
    f = open('big.file', 'w')
    f.close()
    f = open('big.file', 'r+')
    m = mmap(f.fileno(), 10**9)

And now big.file is (about) 1 gigabyte. On Linux, though, this will return ValueError: mmap length is greater than file size.

Is there a way to get the same behavior on Linux as with Windows? That is, to be able to increase a file's size using mmap?

chew socks
  • 1,406
  • 2
  • 17
  • 37

1 Answers1

6

On POSIX systems at least, mmap() cannot be used to increase (or decrease) the size of a file. mmap()'s function is to memory map a portion of a file. It's logical that the thing you request to map should actually exist! Frankly, I'm really surprised that you would actually be able to do this under MS Windows.

If you want to grow a file, just ftruncate() it before you mmap() it.

Celada
  • 21,627
  • 4
  • 64
  • 78
  • Oh that's useful cause you can shorten a file too. Now I wish Windows had `ftruncate()`! – chew socks Jun 30 '13 at 19:19
  • Windows does not have `ftruncate()`? Well, that surprises me too :-) Yet surely it must have *some* equivalent method of growing or shrinking files. – Celada Jun 30 '13 at 19:24
  • I should say that I'm basing that statement off the Python Docs, not off any Windows documentation. – chew socks Jun 30 '13 at 20:04
  • 4
    For anyone reading in the future, file objects also have a `truncate()` method which works like `os.ftruncte()` accept that it is available on both Windows and Unix. This is useful because Pypy has Unix like behavior for growing a memory mapped file, even on Windows. – chew socks Jul 10 '13 at 03:38
  • Is there a way to memory map a file, then grow the file and then remap the file? I'm wondering if databases uses memory mapping and is also capable of growing the underlying file. – CMCDragonkai Apr 12 '18 at 03:08
  • @chewsocks: And for people reading further in the future, [Python 3.5 added support for `os.ftruncate`](https://docs.python.org/3/library/os.html#os.ftruncate) (and `os.truncate` as well), so you can use whichever one makes sense (`truncate` method if you've already got a file object, `os.truncate`/`os.ftruncate` if you've only got a file descriptor). – ShadowRanger Sep 13 '19 at 16:36
  • `ftruncate` then remap would be considered as thread/process safe? – TomSawyer May 14 '20 at 19:19