15

I'm trying to share objects between the parent and child process in Python. To play around with the idea, I've created a simple Python script:

from multiprocessing import Process
from os import getpid

import psutil

shared = list(range(20000000))

def shared_printer():
    mem = psutil.Process(getpid()).memory_info().rss / (1024 ** 2)
    print(getpid(), len(shared), '{}MB'.format(mem))

if __name__ == '__main__':
    p = Process(target=shared_printer)
    p.start()
    shared_printer()
    p.join()

The code snippet uses the excellent psutil library to print the RSS (Resident Set Size). When I run this on OSX with Python 2.7.15, I get the following output:

(33101, 20000000, '1MB')
(33100, 20000000, '626MB')

When I run the exact same snippet on Ubuntu (Linux 4.15.0-1029-aws #30-Ubuntu SMP x86_64 GNU/Linux), I get the following output:

(4077, 20000000, '632MB')
(4078, 20000000, '629MB')

Notice that the child process' RSS is basicall 0MB on OSX and about the same size as the parent process' RSS in Linux. I had assumed that copy-on-write behavior would work the same way in Linux and allow the child process to refer to the parent process' memory for most pages (perhaps except the one storing the head of the object).

So I'm guessing that there's some difference in the copy-on-write behavior in the 2 systems. My question is: is there anything I can do in Linux to get that OSX-like copy-on-write behavior?

chhabrakadabra
  • 430
  • 3
  • 10
  • Have a look at the top answer to that question https://stackoverflow.com/questions/1268252/possible-to-share-in-memory-data-between-2-separate-processes – Tomasz Swider Dec 26 '18 at 18:58
  • 4
    Might just be a matter of how the OS reports RSS values? In addition to looking at the process's memory, check free memory before/after the fork and see if 600MB disappears. – Perette Dec 27 '18 at 17:53
  • 1
    see this answer: https://stackoverflow.com/a/21049737/4201810 – georgexsh Dec 27 '18 at 20:47
  • All your libs are probably statically linked in CPython for MAC and not on Ubuntu ? Look at https://superuser.com/questions/1162037/what-does-precompile-standard-library-option-mean-in-python-installation – mat.viguier Dec 27 '18 at 21:06
  • Is the result consistent? Just OOC, what happens if you explicitly `import gc` and add `gc.disable()` just before creating the `Process`? It's possible cycle collection runs are messing with reference counts and forcing more copy-on-write (not likely to be consistent, but worth ruling out). Also try adding `print(multiprocessing.get_start_method())` to be sure there isn't some weirdness with one OS changing the default start method from the non-Windows default of `fork` to `forkserver` or `spawn` for some reason. – ShadowRanger Dec 27 '18 at 23:17
  • @mat.viguier: Your comment and your link seem unrelated. Static linkage is about C libraries, pre-compiling modules just changes whether the `.pyc` files are generated at install time or at runtime, but either way, it shouldn't affect the behavior of `fork`. – ShadowRanger Dec 27 '18 at 23:26
  • 1
    @ShadowRanger my first thought was also about the GC, but the behavior is consistent regardless. I'm thinking it has to do with how XNU systems reports memory. See [this question](https://stackoverflow.com/q/14789672/5747944) on `host_statistics` (which is used by `psutil`) and the answer as to why its results are inconsistent across platforms. – sytech Dec 29 '18 at 03:49
  • @ShadowRanger : Off course you're right. It's related to the way the CPython on MAC and the CPython on Linux are using the underlying library. If OP is sure the libraries are the same, my comment is completely out of the scope. If not, it is possible that the copy-on-write (as itself, not just a fork) just doesn't work on one of the tow hosts ... – mat.viguier Dec 31 '18 at 09:20

1 Answers1

3

So I'm guessing that there's some difference in the copy-on-write behavior >in the 2 systems. My question is: is there anything I can do in Linux to >get that OSX-like copy-on-write behavior?

The answer is NO. Behind the command psutil.Process(getpid()).memory_info().rss / (1024 ** 2) the OS uses the UNIX command $top [PID] and search for the field RES. Which contains the non-swapped physical memory a task has used in kb. i.e. RES = CODE + DATA.

IMHO, these means that both OS uses different memory managers. So that, it's almost impossible to constrain how much memory a process uses/needs. This is a intern issue of the OS. In Linux the child process has the same size of the parent process. Indeed, they copy the same stack, code and data. But different PCB (Process Control Block). Therefore, it is impossible to get close to 0 as OSX does. It smells that OSX does not literally copy the code and data. If they are the same code, it will make pointer to the data of the parent process.

PD: I hope that would help you!

Agapito Gallart Bernat
  • 343
  • 2
  • 11
  • 1
    Both Linux and OSX should be using `fork` with the same copy-on-write semantics for the memory in existence at the moment the child forks off from the parent. I'd be more inclined to suspect a difference in how OSX *counts* memory, not in how much it uses. Both of them should have almost no *real* memory usage immediately after the fork, but Linux may report the memory use assuming all copy-on-write pages end up copied (thanks to CPython's reference counted+cycle collector design, most probably will), while OSX only reports unique private, non-copy-on-write, pages. – ShadowRanger Dec 27 '18 at 23:20