2

My program loads a lot of data at start up and then calls debug.FreeOSMemory() so that any extra space is given back immediately.

loadDataIntoMem()
debug.FreeOSMemory()

after loading into memory , htop shows me the following for the process

 VIRT    RES     SHR
 11.6G   7629M   8000

But a call to runtime.ReadMemStats shows me the following

Alloc         5593336608   5.3G
BuckHashSys   1574016      1.6M
HeapAlloc     5593336610   5.3G
HeapIdle      2607980544   2.5G
HeapInuse     7062446080   6.6G
HeapReleased  2607980544   2.5G
HeapSys       9670426624   9.1G
MCacheInuse   9600         9.4K
MCacheSys     16384        16K
MSpanInuse    106776176    102M
MSpanSys      115785728    111M
OtherSys      25638523     25M
StackInuse    589824       576K
StackSys      589824       576K
Sys           10426738360  9.8G
TotalAlloc    50754542056  48G
  1. Alloc is the amount obtained from system and not yet freed ( This is resident memory right ?) But there is a big difference between the two.
  2. I rely on HeapIdle to kill my program i.e if HeapIdle is more than 2 GB, restart - in this case it is 2.5, and isn't going down even after a while. Golang should use from heap idle when allocating more in the future, thus reducing heap idle right ?
  3. If assumption 1 is wrong, which stat can accurately tell me what the RES value in htop is.
  4. What can I do to reduce the value of HeapIdle ?

This was tried on go 1.4.2, 1.5.2 and 1.6.beta1

Abhishek Shivanna
  • 73
  • 7
  • I looked at [memory decrease golang question](http://stackoverflow.com/questions/27497512/why-does-not-the-memory-decrease-in-golang) but it only says dont depend on FreeOSMemory or runtime.GC(), what else can I do ? – Abhishek Shivanna Dec 23 '15 at 08:13
  • 1
    As [a very skillful man](https://en.m.wikipedia.org/wiki/Donald_Knuth) once said: ["Premature optimization is the root of all evil."](http://c2.com/cgi/wiki?PrematureOptimization). With just slightly under 8 Gb of physical and below 12 Gb virtual memory allocated, you shouldn't bother too much. Hint: there is a difference between "sum of memory allocated in the past" and "currently used". – Markus W Mahlberg Dec 23 '15 at 11:17
  • @JimB I'm storing alot of data in memory for faster access, and since I want to avoid allowing the os to start swapping pages, since other processes also use the same server, I thought of killing it when it goes into a bad state. Btw - How long before it goes back to the OS ? Its been ten hours and HeapIdle is still 2.5G . It seems like this is never going back to the os. – Abhishek Shivanna Dec 23 '15 at 18:48
  • @AbhishekShivanna have you ever got to the bottom of it? Were you able to reduce the HeapIdle? – maxcnunes Mar 12 '19 at 13:18

1 Answers1

1

The effective memory consumption of your program will be Sys-HeapReleased. This still won't be exactly what the OS reports, because the OS can choose to allocate memory how it sees fit based on the requests of the program.

If your program runs for any appreciable amount of time, the excess memory will be offered back to the OS so there's no need to call debug.FreeOSMemory(). It's also not the job of the garbage collector to keep memory as low as possible; the goal is to use memory as efficiently as possible. This requires some overhead, and room for future allocations.

If you're having trouble with memory usage, it would be a lot more productive to profile your program and see why you're allocating more than expected, instead of killing your process based on incorrect assumptions about memory.

JimB
  • 104,193
  • 13
  • 262
  • 255
  • Thanks for the explanation. I get and put to a cache, but to avoid hassles of locks, I make clones before giving a value. Since this cache has alot of entries, the creation and deletion of all these clones is putting a load on the gc. I wanted to call `debug.FreeOSMemory()` periodically as I don't want the os to start swapping pages since that will affect other processes on the same server. – Abhishek Shivanna Dec 23 '15 at 18:52
  • @AbhishekShivanna: just like the docs for FreeOSMemory say, memory is returned by the runtime to the OS in a background task (called the heap scavenger). That can take 5-10 minutes with the current settings. You can see what's happening with GC and the scavenger by running with gctrace. – JimB Dec 23 '15 at 19:26