1

This is a simple program to decompress the string, I just running a loop to show that memory usage increases and the memory used never get released.

Memory is not getting released even after 8hr also

Package for decompressing string: https://github.com/Albinzr/lzGo - (simple lz string algorithm)

I'm adding a gist link since the string used for decompressing is large

Source Code: Code

enter image description here

Activity Monitor enter image description here

I'm completely new to go, Can anyone tell me how I can solve the memory issue?

UPDATE Jul 15 20 The app still crashes when the memory limit is reached Since it only uses 12mb - 15mb this should not happen!!

Albi
  • 1,705
  • 1
  • 17
  • 28
  • 2
    It might take a while for the runtime to release memory to the OS, see [Cannot free memory once occupied by bytes.Buffer](https://stackoverflow.com/questions/37382600/cannot-free-memory-once-occupied-by-bytes-buffer/37383604#37383604). There's not enough info here to draw the conclusion you did. The library you used might also maintain an internal cache. – icza Jul 10 '20 at 07:26
  • @icza I have kept this program running more than 8hr still there is no memory release, I have listed the entire code here, please read the question completely, – Albi Jul 10 '20 at 07:43
  • @icza I had already checked all the stuff you mentioned already, and as I mentioned I'm new to go and it will be great if you can help me out – Albi Jul 10 '20 at 07:47
  • For code-review, put your code here: https://codereview.stackexchange.com/ – shmsr Jul 10 '20 at 07:55
  • @shmsr I may be wrong, but code review is to improve your codebase and coding standards, but here I'm having an issue with the memory, I want to figure out why and how this is happening and solve it before going to the code review stage – Albi Jul 10 '20 at 07:59
  • 1
    this is not the way to check for memory usage. See expvar and for a quick setup expvarmon. Learn to read the metrics, understand them, then draw conclusion. –  Jul 10 '20 at 08:10
  • 2
    You launch a webserver and your profile shows your `main` occupies 12 MB RAM? That's not really a memory leak. The 182 MB you see in your Activity Monitor might be just for the profiler. I don't see an issue here. – icza Jul 10 '20 at 08:12
  • @icza that's also confusing thing I'm also seeing, If you have time please try running the code I linked and you will get to know what I'm saying – Albi Jul 10 '20 at 08:21
  • @mh-cbon I have done things possible to the current knowledge I have, Just started go a month back, If you can check out the code or point me in the right direction it will be great – Albi Jul 10 '20 at 11:37
  • @icza I ran the code without debug or profiler still, it shows high memory usage and never releases. I even tried it with a container that uses ubuntu just to check if it was any os related issue but in that also I saw that same high memory usage and no release of captured memory. Let me know if there are any other tests I can do to figure out the issue, Thanks – Albi Jul 11 '20 at 05:55
  • 1
    I tried running your code in manjaro. RES -- Resident size is 25M. – Billy Yuan Jul 17 '20 at 02:41
  • @BillyYuan The issue I'm seeing is that the go app takes large memory when the loop size increases and at certain point when there is no more free memory it crashes rather than using the allocated memory – Albi Jul 17 '20 at 11:05

1 Answers1

6

There is a lot going on here.

First, using Go version 1.14.2 your program works fine for me. It does not appear to be leaking memory.

Second, even when I purposely created a memory leak by increasing the loop size to 100 and saving the results in an array, I only used about 100 MB of memory.

Which gets us to Third, you should not be using Activity Monitor or any other operating system level tools to be checking for memory leaks in a Go program. Operating System memory management is a painfully complex topic and the OS tools are designed to help you determine how a program is affecting the whole system, not what is going on within the program.

Specifically, macOS "Real Memory" (analogous to RSS, Resident Set Size) includes memory the program is no longer using but the OS has not taken back yet. When the garbage collector frees up memory and tells the OS it does not need that memory anymore, the OS does not immediately take it back. (Why it works that way is way beyond the scope of this answer.) Also, if the OS is under Memory Pressure, it can take back not only memory the program has freed, but it can also take back (temporarily) memory the program is still using but has not accessed "recently" so that another program that urgently needs memory can use it. In this case, "Real Memory" will be reduced even if the process is not actually using less memory. There is no statistic reported by the operations system that will help you here.

You need to use native Go settings like GODEBUG=gctrace=1 or tools like expvar and expvarmon to see what the garbage collector is doing.

As for why your program ran out of memory when you limited it, keep in mind that by default Go builds a dynamically linked executable and just reading in all the shared libraries can take up a lot of memory. Try building your application with static linking using CGO_ENABLED=0 and see if that helps. See how much memory it uses when you only run 1 iteration of the loop.

Old Pro
  • 24,624
  • 7
  • 58
  • 106