Allocated a lot of memory in Go. How to fix?

Question

Several hundred MB of memory is allocated for 50 requests of 5 MB. Memory is allocated and is no longer released. How can I clear my memory? Why can this happen?

I've tried on Ubuntu on my home pc and on VPS

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "time"
)

func main() {
    fmt.Println("start")

    for i := 0; i < 50; i++ {
        go func() {
            DoRequest()
        }()
        time.Sleep(10 * time.Millisecond)
    }

    time.Sleep(10 * time.Minute)
}

func DoRequest() error {
    requestUrl := "https://blockchain.info/rawblock/0000000000000000000eebedea046425bd54626e6c56eb032e66e714d0141ea6"

    req, err := http.NewRequest("GET", requestUrl, nil)

    if err != nil {
        return err
    }

    req.Header.Set("user-agent", "free")

    httpClient := &http.Client{
        Timeout: time.Second * 10,
    }

    resp, err := httpClient.Do(req)

    if resp != nil {
        defer resp.Body.Close()
    }

    if err != nil {
        return err
    }

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return err
    }

    fmt.Println("bodylen", len(body))

    return nil
}

Allocated somewhere 400MB

Allocating it is normal. Holding it until there is enough memory pressure on the system for the OS to reclaim it is normal. What is your question? — Adrian, Jun 10 '19 at 14:03

colm.anseo · Answer 1 · 2019-06-10T22:47:22.790

8

You are creating an http client for each go-routine.

Http client is designed to be create once & used many times. They are go-routine safe. They allow for connection reuse & other efficiency savers.

Create the http client once in main (instead of in your go-routine) & then pass this single reference to all of your 50 go-routines.

Edit: Also, while it may not make a practical difference in your case, the order for a request is usually like so:

resp, err := httpClient.Do(req)
if err != nil {
        return err // check error first
}
defer resp.Body.Close() // no error - so resp will *NOT* be nil - so this is safe

Edit 2: As @Adrian has mentioned: go's garbage collection is not instantaneous - nor should it be - as it is an expensive operation. If you no longer need a block of memory - simply don't reference it anymore. Let the GC do its job, so you can focus on yours!

If you're curious about the evolution of go's GC:

https://blog.golang.org/ismmkeynote (heavy on the technical side)
What kind of Garbage Collection does Go use?

edited Jun 10 '19 at 22:47

answered Jun 10 '19 at 03:57

colm.anseo

19,337
4
43
52

Not help. Thanks for the reply – xxxcvv Jun 10 '19 at 12:04
I have 256MB memory per server container. And i need to handle concurrently many requests with big response sizes. And now the memory does not go away, it freezes and is not released. all the memory gets filled up and cannot be released more. Even 512MB is not enough, checked – xxxcvv Jun 11 '19 at 02:26
Did you fix the order of error checking described above? Keying off of `resp != nil` may cause unpredictable results, if a resp body is not a valid response due to error. – colm.anseo Jun 11 '19 at 02:39

Manish Champaneri · Accepted Answer · 2019-06-11T11:08:08.773

 for i := 0; i < 50; i++ {
        go func() {
            DoRequest()
        }()
        time.Sleep(10 * time.Millisecond)
    }

Never create go-routines like this. Always make sure you create go-routines the way it not fill large ( all ) memory in any case ( including worst case )

Simple solution is control the count of go-routines can spawned ( or running ) at time.

You can pre-calculate memory to be occupied in worst case by multiplying max-number of go-routines you want to run at a time and max-memory can be used by one go-routine.

You can control instances of go-routines by using channles.

Refer first answer of this stackoverflow question Always have x number of goroutines running at any time

Always use balanced solution between perforamce and required resources.

Update June 11,2019

Here is example go program https://play.golang.org/p/HovNRgp6FxH

Allocated a lot of memory in Go. How to fix?

2 Answers2