-1

I just ported some code from C/C++ to Go, it is a microservice. It works well, even faster than in C/C++. But I have a problem with memory.

When my program starts it will allocate about 4.5GB RAM and will fill it with data from disc also processing data while loading, then it will run for days(hopefully months) serving the requests from RAM. Unfortunately after the processing and placement of data in RAM is finished, extra 3.5 GB of RAM remains allocated by Go. I do not do any deallocations, only allocations, I do not think my program really uses 8 GB at any point, so I think Go just acquires extra RAM because it "feels" I might need more soon, but I will not.

I read that Go does not allow any functionality to deallocate unused RAM to return it to system. I want to run more services on the same machine, saving as much of RAM as possible, so wasting almost as much as I really use feels wrong.

So how do I keep the memory footprint compact avoiding those empty 3.5 GB being allocated by Go?

exebook
  • 32,014
  • 33
  • 141
  • 226
  • See related questions: [Golang - Cannot free memory once occupied by bytes.Buffer](http://stackoverflow.com/questions/37382600/golang-cannot-free-memory-once-occupied-by-bytes-buffer/37383604#37383604); and [FreeOSMemory() in production](http://stackoverflow.com/questions/42345060/freeosmemory-in-production/42345554#42345554). – icza Mar 08 '17 at 11:46
  • How do you know that memory isn't being used, or if it's really allocated at all and not just virtual size? What does the gctrace output look like? What are the actual memstats showing what's allocated? – JimB Mar 08 '17 at 11:49
  • "4.5GB RAM and will fill it with data from disc also processing data while loading" hints at that you are not only allocating a flat `[4.5e6]byte` but that your data structure is more complicated. It is literally impossible to suggest anything you could do to reduce the memory footprint of your application with the few information you gave. – Volker Mar 08 '17 at 11:50
  • @exebook How are you allocating the memory? – Ankur Mar 08 '17 at 14:33

2 Answers2

1

Speaking of virtual memory (See "Go Memory Management" from Povilas Versockas, and RSS vs. VSZ), don't try your program with Go 1.11.

See golang/go issue 28114: "runtime: programs compiled by 1.11 allocate an unreasonable amount of virtual memory".
Also discussed here.

Possibly related to:

  • CL 85888: runtime: remove non-reserved heap logic
  • issue 10460: runtime: 512GB memory limitation

Possible workaround: golang/go issue 28081

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
0

Most likely that is virtual memory that Go is using, not actual committed pages of RAM.

DeferPanic blog post: "Understanding Go Lang Memory Usage" (2014)

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
Jonathon Reinhart
  • 132,704
  • 33
  • 254
  • 328