Allocation error: runtime: out of memory

Question

I wrote this code:

package main

import (
    "log"
)

func main() {
    var c []int64
    for i := 0; i < 100; i++ {
        c = make([]int64, 10000000000)
        log.Println(len(c))
    }
}

This code runs out of memory: fatal error: runtime: out of memory.

In every iteration, c will be assigned a new slice. So the previous slice is not reachable. Why doesn't the GC appear to collect unreachable memory?

Related: [Golang - Cannot free memory once occupied by bytes.Buffer](https://stackoverflow.com/questions/37382600/golang-cannot-free-memory-once-occupied-by-bytes-buffer/37383604#37383604). — icza, Nov 23 '17 at 07:38
@icza: This is not a duplicate. This is a case of over allocation, an attempt to allocate 80GB in a single allocation, which immediately fails: `fatal error: runtime: out of memory`. Officially, to be a duplcate, the answer must be the same. In this case, the answer is make reasonable sized allocations. In your answer, it's an explanation of GC free memory mechanisms, which, while interesting, doesn't solve this problem. . — peterSO, Nov 23 '17 at 13:32
@peterSO That's why I just wrote "Related" and not "duplicate". — icza, Nov 23 '17 at 13:35

peterSO · Answer 1 · 2017-11-23T15:09:02.333

Each c = make([]int64, 10000000000 is attempting to allocate 80GB (8 * 10,000,000,000 bytes) of memory. Use a reasonable sized allocation (relative to the size of your real memory) and everything works as expected. For example,

package main

import (
    "fmt"
    "log"
    "runtime"
)

func main() {
    var ms runtime.MemStats
    runtime.ReadMemStats(&ms)
    fmt.Println(ms.TotalAlloc, ms.Alloc)
    var c []int64
    for i := 0; i < 100; i++ {
        c = make([]int64, 400000000)
        log.Println(len(c), i)
    }
    runtime.ReadMemStats(&ms)
    fmt.Println(ms.TotalAlloc, ms.Alloc)
}

Output:

67032 67032
2017/11/23 01:13:08 400000000 0
2017/11/23 01:13:09 400000000 1
2017/11/23 01:13:09 400000000 2
2017/11/23 01:13:09 400000000 3
2017/11/23 01:13:10 400000000 4
<<SNIP>>
2017/11/23 01:13:43 400000000 95
2017/11/23 01:13:43 400000000 96
2017/11/23 01:13:43 400000000 97
2017/11/23 01:13:44 400000000 98
2017/11/23 01:13:44 400000000 99
320000171152 88168

You tried to allocate 80,000,000,000 bytes for c. I reduced it to something more reasonable 3,200,000,000 bytes. The loop allocated 100 times for a total of 320,000,171,152 bytes, which the garbage collector handled by reusing memory. The GC is working.

You change the size of "c". I mean every iteration the memory size is ok. But if loop 100 times, memory is not enough. In every iteration, the last slice is not reachable and I think it should be collected by the GC. But it is not. So why? — z xt, Nov 23 '17 at 05:21

Allocation error: runtime: out of memory

1 Answers1