-1

I am working with a very large map of pointer to struct. It is growing over the lifetime of the program (I use it as a buffer) and I wrote a function that is supposed to reduce it size when it is called.

type S struct {
 a uint32
 b []uint32
}

s := make(map[uint32]*S)

for k, v := range s {
  delete(s, k)
  s[k] = &S{a: v.a}
}

I remove b from every element of the map, so I expected the size of the map in memory to shrink (b is a slice of length > 10). However the memory is not freed, why?

Kevin P
  • 273
  • 1
  • 3
  • 13

2 Answers2

1

The size of the map value &S, a pointer, is the same irrespective of the capacity of slice b.

package main

import (
    "fmt"
    "unsafe"
)

type S struct {
    a uint32
    b []uint32
}

func main() {
    size := unsafe.Sizeof(&S{})
    fmt.Println(size)

    size = unsafe.Sizeof(&S{b: make([]uint32, 10)})
    fmt.Println(size)

    s := make(map[uint32]*S)

    for k, v := range s {
        delete(s, k)
        s[k] = &S{a: v.a}
    }
}

Output:

8
8

A slice is represented internally by

type slice struct {
    array unsafe.Pointer
    len   int
    cap   int
}

When you set &S{a: v.a} you set b to initial values: array to nil and len and cap to zero. The memory formerly occupied by the underlying array is returned to the garbage collector for reuse.

peterSO
  • 158,998
  • 31
  • 281
  • 276
  • 1
    Yes, sorry in my question I was talking about the total size in memory (so the size of the map + to all the struct it points to). What you describe is indeed the behaviour I expected writing my code. However the garbage collector does not seem to free/allow for reuse the formerly occupied memory by the strucs. – Kevin P Jun 24 '19 at 06:45
1

The map size is bounded to the maximum size it had at any point. Because you store pointers (map[uint32]*S) and not values the deleted objects will get garbage collected eventually but usually not immediately and that why you don’t see it in top/htop like monitors.

The runtime is clever enough and reserves memory for future use if the system is not under pressure or low on resources.

See https://stackoverflow.com/a/49963553/1199408 to understand more about memory.

In your example you don't need to call delete. You will achieve what you want just by clearing the slice in the struct.

type S struct {
 a uint32
 b []uint32
}

s := make(map[uint32]*S)

for k, v := range s {
  v.b = []uint32{}
}
georgeok
  • 5,321
  • 2
  • 39
  • 61
  • Isn't the size of a slice also bounded to the maximum it had? This is the reason why I performed the delete operation: so that the size of the slice in `s` is also reset – Kevin P Jun 24 '19 at 06:40
  • No because it’s a pointer which is the reference to the struct instance – georgeok Jun 24 '19 at 06:53
  • Yes but I am interested in the global memory usage, not only the map size. So each element of the map points to a struct which contains a slice. – Kevin P Jun 24 '19 at 07:37
  • Yes, the deleted instances will eventually get garbage collected but there is a chance that the Go runtime will not immediately return the memory to the OS. – georgeok Jun 24 '19 at 07:45
  • My question is exactly about this issue: My program gets killed by the system (out of memory) before the gc has time to release the unused memory – Kevin P Jun 24 '19 at 08:20
  • That's a new thing. Use the environment variables from https://golang.org/pkg/runtime/ or pprof https://golang.org/pkg/runtime/pprof/ to monitor the memory size and edit your question with the finding to give us more information. – georgeok Jun 24 '19 at 08:27
  • 1
    `v.b = v.b[:0]` frees zero memory. The same underlying array is still used by the slice. It just means the slice won't have to grow the next time it's appended to. `v.b := []uint32{}` would actually free memory. – Adrian Jun 24 '19 at 13:43
  • That's a good point. I've edited the answer adopting this change. – georgeok Jun 24 '19 at 13:52