2

I have a set of python processes that share a large object (sharing is done by forking the processes after initializing the object)

I notice a strange memory leak:

  • the process memory (VSZ and RSS) hardly change
  • the total system memory increases steadily

My guess is the shared object does change (it's 'logically' readonly, but it's possible that some internal private variables change even when just reading from it) - which causes memory pages to be copied

is there a way to verify this?

Ophir Yoktan
  • 8,149
  • 7
  • 58
  • 106
  • What kind of object? A list? A dict? A pandas Dataframe? Do you have access to the sourcecode? (of the object) – MegaIng Jun 26 '18 at 07:22
  • a pandas dataframe – Ophir Yoktan Jun 26 '18 at 07:31
  • https://stackoverflow.com/questions/14224068/memory-leak-using-pandas-dataframe is this related? If it's that, than that is probably OS specific and you might try to manually trigger the garbage collection by doing `gc.collect()` after a chunck of your work is done. – user1767754 Jun 26 '18 at 07:48

2 Answers2

1

To answer your specific question "is there a way to verify this?", if I understand it correctly, you can do the following if you want to see whether there are any changes associated with the pages that contain the large object.

1) Determine the address of your "large shared object" and the address of where that object ends.

2) If the start address is not on a 4K page boundary, round the start address down to the page boundary before where the object starts.

3) If the end address is not on a 4K boundary, round the end address up to the page boundary after where the object ends.

4) Dump that memory range for the process and all its children to separate files and compare them.

However, I think that Will os.fork() use copy on write or do a full copy of the parent-process in Python? may already provide an explanation for at least some of the writing that is requiring copying. Specifically, python objects are reference counted and your child processes will be altering reference counts.

Have you considered using python's threading rather than creating child processes?

Tim Boddy
  • 1,019
  • 7
  • 13
0

You can give the following a try:

dtrace -n wp_page_copy:entry'/pid == 11111/ { @counts[ustack()] = count(); }'

Not sure if wp_page_copy is restricted to only being used during copy-on-write. I seem to recall reading that it is, but I wouldn't bet on it without confirming. You'll want to change 11111 to your python pid.

interestedparty333
  • 2,386
  • 1
  • 21
  • 35