2

I have a Go program that allocates lots of maps and slices. Generally a lot of usage, allocation overhead etc. I run it, it loads a lot of data in, and then I query it with a web service.

After I leave it running, when it's read in all its data and isn't doing any queries (i.e should be stable) I see memory fluctuations. Recently it's reported: 5.42 GB, 5.01 GB and 4.3 GB of real memory. That's a massive fluctuation.

I have about 150 million objects (slices hanging off the main hashtable). That's a lot of little objects. I expect a little fluctuation (although I would never expect memory to increase when no new objects are being allocated and the main thread/s is blocking on a socket).

Possible explanations are

  • the overhead of lots of small allocations just multiplies any natural fluctuation
  • some code is allocating objects (although I can't see how)
  • the Go GC is doing its own paging (?)
  • I'm using Mac OS, and it's at fault somehow

Is this amount of fluctuation normal / expected?

Joe
  • 46,419
  • 33
  • 155
  • 245
  • How fast did this fluctuate? Can you put timestamps on those memory usage measurements? – joshlf Jan 11 '14 at 17:12
  • Over the course of 10 minutes. I can't put timestamps on it, but could replicate if it's important. – Joe Jan 11 '14 at 17:18
  • No, I think "minute scale" is granular enough. – joshlf Jan 11 '14 at 17:21
  • Have you tried using the [profiler](http://blog.golang.org/profiling-go-programs) on your code? It can help you identify areas where you have lots of allocations so that you can change the code to avoid them. – markc Jan 11 '14 at 19:39
  • I could, yes. I know I'm allocating a lot of objects and I know I could be better about allocation. My question is more about should I expect the memory to fluctuate after allocation? – Joe Jan 12 '14 at 00:21
  • The fluctuation is up and down or only down? Garbage Collector in Go runs every two minutes when the software is idle. And memory is returned to the OS five minutes after it is colleced by the garbage collector. In a worst-case scenario, memory is returned to the OS after 7 minutes (2 to be GCed, 5 to be freed). – siritinga Jan 12 '14 at 08:44
  • Yes, up and down. My questions are about the ups with no new allocations. – Joe Jan 12 '14 at 11:43

2 Answers2

1

The go-runtime doesn't immediately release unused memory to the OS (it might be needed again soon). So looking at the OS-level, you see only a part of the overall picture. Using http://golang.org/pkg/runtime/#ReadMemStats you can see another part of the picture.

pkg/runtime/malloc.goc shows the freelist, and pkg/runtime/mgc0.c shows the garbage collector.

If memory-usage goes down in a stable situation that seems normal, after loading finishes, you can force a GC, and you might want to print the Memstats regularly for more info.

user607139
  • 356
  • 2
  • 4
  • Thanks, I'll look at the memstats if this happens again. Do you have any idea about why memory consumption increases? – Joe Jan 12 '14 at 18:54
0

The fluctuations are likely due to the amount of garbage your program is creating that the garbage collector has to eventually collect. The frequency of the fluctuation is going to depend on how much / how often you are creating garbage and when the garbage collector collects it.

Whether a variable is allocated to the stack or the heap is determined by the compiler. Generally pointers, maps, and slices can be allocated to the heap, but this only happens if the compiler's escape analysis determines that a variable escapes. Anything that is allocated to the heap will need to be garbage collected.

Even though Go handles the stack vs heap details, creating as little garbage as possible can be of great benefit. You can read about an extreme case where the garbage collector paused for 10 seconds. The Go garbage collector isn't perfect, but it is improving. The more it improves less you'll have to worry about it. But you should at least be aware of it.

You can run the following to determine what the compiler will allocate to the heap:

go build -gcflags=-m program.go

You may be surprised by what actually gets allocated to the heap. For example, even if you use a bytes.Buffer locally it still gets allocated to the heap, due to bytes.Buffer.buf being re-sliced. Regardless of whether that's supposed to happen or not, there may be situations where you think you're not creating any garbage, but in reality you are.

Community
  • 1
  • 1
Luke
  • 13,678
  • 7
  • 45
  • 79