Memory Management in Ruby

Question

I puzzled by some behaviour of ruby and how it manages memory.

I understand the Ruby GC (major or minor) behaviour i.e if the any objects count goes above there threshold value or limit (i.e heap_available_slots,old_objects_limit, remembered_shady_object_limit, malloc_limit). Ruby run/trigger a GC(major or minor).

And after GC if it can't find the enough memory Ruby allocate (basically malloc I assuming) more memory for the running program.

Also, It's a known fact that by does release back memory to the OS immediately.

Now ..

What I fail to understand how come Ruby releases memory (back to the OS) without triggering any GC.

Example

require 'rbtrace'

index = 1
array = []
while(index < 20000000) do 
   array << index
   index += 1
end


sleep 10
print "-"
array=nil
sleep

Here is my example. If run the above code on ruby 2.2.2p95. htop display the RSS count of the process (test.rb PID 11483) reaching to 161MB.

GC.stat (captured via rbtrace gem) look like (pay close attention to attention to GC count)

rbtrace -p 11843 -e '[Time.now,Process.pid,GC.stat]'

[Time.now,Process.pid,GC.stat]
=> [2016-07-27 13:50:28 +0530, 11843, 
{
  "count": 7,
  "heap_allocated_pages": 74,
  "heap_sorted_length": 75,
  "heap_allocatable_pages": 0,
  "heap_available_slots": 30162,
  "heap_live_slots": 11479,
  "heap_free_slots": 18594,
  "heap_final_slots": 89,
  "heap_marked_slots": 120,
  "heap_swept_slots": 18847,
  "heap_eden_pages": 74,
  "heap_tomb_pages": 0,
  "total_allocated_pages": 74,
  "total_freed_pages": 0,
  "total_allocated_objects": 66182,
  "total_freed_objects": 54614,
  "malloc_increase_bytes": 8368,
  "malloc_increase_bytes_limit": 33554432,
  "minor_gc_count": 4,
  "major_gc_count": 3,
  "remembered_wb_unprotected_objects": 0,
  "remembered_wb_unprotected_objects_limit": 278,
  "old_objects": 14,
  "old_objects_limit": 10766,
  "oldmalloc_increase_bytes": 198674592,
  "oldmalloc_increase_bytes_limit": 20132659
}]
*** detached from process 11843

GC count => 7

Approximately 25 minutes later. Memory has drop down to 6MB but GC count is still 7.

[Time.now,Process.pid,GC.stat]
=> [2016-07-27 14:16:02 +0530, 11843, 
{
  "count": 7,
  "heap_allocated_pages": 74,
  "heap_sorted_length": 75,
  "heap_allocatable_pages": 0,
  "heap_available_slots": 30162,
  "heap_live_slots": 11581,
  "heap_free_slots": 18581,
  "heap_final_slots": 0,
  "heap_marked_slots": 120,
  "heap_swept_slots": 18936,
  "heap_eden_pages": 74,
  "heap_tomb_pages": 0,
  "total_allocated_pages": 74,
  "total_freed_pages": 0,
  "total_allocated_objects": 66284,
  "total_freed_objects": 54703,
  "malloc_increase_bytes": 3248,
  "malloc_increase_bytes_limit": 33554432,
  "minor_gc_count": 4,
  "major_gc_count": 3,
  "remembered_wb_unprotected_objects": 0,
  "remembered_wb_unprotected_objects_limit": 278,
  "old_objects": 14,
  "old_objects_limit": 10766,
  "oldmalloc_increase_bytes": 198663520,
  "oldmalloc_increase_bytes_limit": 20132659
}]

Question: I was under the impression that Ruby Release memory whenever GC is triggered. But clearly that not the case over here.

Anybody can provide a detail on how (as in who triggered the memory releases surely its not GC.) the memory is released back to OS.

OS: OS X version 10.11.12

Casper · Accepted Answer · 2018-02-14T16:35:09.313

4

You are correct, it's not GC that changed the physical memory requirements, it's the OS kernel.

You need to look at the VIRT column, not the RES column. As you can see VIRT stays exactly the same.

RES is physical (resident) memory, VIRT is virtual (allocated, but currently unused) memory.

When the process sleeps it's not using its memory or doing anything, so the OS memory manager decides to swap out part of the physical memory and move it into virtual space.

Why keep an idle process hogging physical memory for no reason? So the OS is smart, and swaps out as much unused physical memory as possible, that's why you see a reduction in RES.

I suspect you would see the same effect even without array = nil, by just sleeping long enough. Once you stop sleeping and access something in the array, then RES will jump back up again.

You can read some more discussion through these:
What is RSS and VSZ in Linux memory management
http://www.darkcoding.net/software/resident-and-virtual-memory-on-linux-a-short-example/
What's the difference between "virtual memory" and "swap space"?
http://www.tldp.org/LDP/tlk/mm/memory.html

edited Feb 14 '18 at 16:35

answered Jul 27 '16 at 09:40

Casper

33,403
4
84
79

I wish I can read more about this any link or source would be great. – Viren Jul 27 '16 at 09:54
the VIRTUAL has stayed constant. Also I how do I check what is the VIRTUAL memory current in used. I guess it's the SWAP column in top that specific that right? – Viren Jul 27 '16 at 09:56
Sorry I noticed you're using OSX, but it's similar to Linux. On Linux you get detailed total memory statistics for example `cat /proc/meminfo`. It's not as simple as just looking at Swap, because the OS uses some memory for disk caching also. I added some links to the answer where you can read more. – Casper Jul 27 '16 at 10:22
http://apple.stackexchange.com/questions/101488/os-xs-free-or-proc-meminfo-equivalent – Casper Jul 27 '16 at 10:24
Before I accept your answer.The way I get this Linux kernel will page out the memory if not in used assuming I wake from the long sleep will the entire page would be swap back in when it resume after long sleep. – Viren Jul 27 '16 at 10:46
Memory is always swapped in and out in pages. So normally if a page was swapped out because it was idle, then if you access any memory inside that page it will be swapped back in to physical memory. This does not mean the entire **array** is swapped in, just the part that is inside the page you are accessing. If you access the whole array then many pages may be swapped back in. It depends on page size and how large the array is to begin with. Hope that helps.. – Casper Jul 27 '16 at 10:56
Thanks you very much. – Viren Jul 27 '16 at 11:06
1

Btw. if you like very technical documentation then you can get a lot of details here: http://www.tldp.org/LDP/tlk/mm/memory.html – Casper Jul 27 '16 at 11:07

Memory Management in Ruby

1 Answers1