2

I am writing a program in Go that depends on regexp. The memory usage as shown in htop is steadily increasing onto the point where the program is crashing. (> 5-7GB)

Yet when I analyze the memory usage with pprof it shows me that only a tiny fraction is actually in use. (~70MB)

I tried manually triggering GC:

go func() {
for range time.Tick(30*time.Second){
    runtime.GC()
    debug.FreeOSMemory()
    runtime.ReadMemStats(&m)
    fmt.Printf("HeapSys: %d, HeapAlloc: %d, HeapIdle: %d, HeapReleased: %d\n", m.HeapSys, m.HeapAlloc,
        m.HeapIdle, m.HeapReleased)
}}()
// Sample output(not from the same run as the other numbers):
// HeapSys: 347308032, HeapAlloc: 123637792, HeapIdle: 194322432, HeapReleased: 0

This shows no effect. My understanding was that golang only requests memory if it needs more and cleans up every once in a while (Running with GCTRACE shows that gc is actually running often) I do not understand why golang continues to request memory when the actually used memory is comparatively small.

Does someone know the underlying cause of the gap and how I can prevent my program from eating memory?

As the graphs are SVGs I cannot embed them:

Possibly related: how to analyse golang memory Does not explain why the memory continues to increase(and is not reused).

Community
  • 1
  • 1
Niels-Ole
  • 129
  • 1
  • 9
  • 2
    The only thing you can "leak" in a sense are goroutines which aren't returning. Look at a stack trace to see what you have running. – JimB Jan 02 '17 at 19:37
  • @JimB Thanks for the hint. I refactored the entire code now to always have a fixed amount of workers that communicate with (buffered) queues. This has stopped the runaway consumption of memory. – Niels-Ole Jan 10 '17 at 17:21

0 Answers0