-2

Go's append() function only allocates new slice data, when the capacity of the given slice is not sufficient (see also: https://stackoverflow.com/a/28143457/802833). This can lead to unexpected behavior (at least for me as a golang newbie):

package main

import (
    "fmt"
)

func main() {

    a1 := make([][]int, 3)
    a2 := make([][]int, 3)
    b := [][]int{{1, 1, 1}, {2, 2, 2}, {3, 3, 3}}
    common1 := make([]int, 0)
    common2 := make([]int, 0, 12) // provide sufficient capacity
    common1 = append(common1, []int{10, 20}...)
    common2 = append(common2, []int{10, 20}...)

    idx := 0
    for _, k := range b {
        a1[idx] = append(common1, k...) // new slice is allocated
        a2[idx] = append(common2, k...) // no allocation
        idx++
    }

    fmt.Println(a1)
    fmt.Println(a2) // surprise!!!
}

output:

[[10 20 1 1 1] [10 20 2 2 2] [10 20 3 3 3]]

[[10 20 3 3 3] [10 20 3 3 3] [10 20 3 3 3]]

https://play.golang.org/p/8PEqFxAsMt

So, what ist the (idomatic) way in Go to force allocation of new slice data or more precisely to make sure that the slice argument to append() remains unchanged?

Community
  • 1
  • 1
kirk
  • 125
  • 1
  • 5
  • 1
    The "problem" is less the behaviour of append but your code which is needless complex. – Volker Nov 18 '15 at 18:37
  • @Volker The example has been extracted out of a bigger context. If I need just to append the shown slices, there indeed is a simpler way :-) – kirk Nov 19 '15 at 08:41

2 Answers2

9

You might maintain a wrong idea of how slices work in Go.

When you append elements to a slice, the call to append() returns a new slice. If reallocation did not happen, both slice values — the one you called append() on and the one it returned back — share the same backing array but they will have different lengths; observe:

package main

import "fmt"

func main() {
    a := make([]int, 0, 10)
    b := append(a, 1, 2, 3)
    c := append(a, 4, 3, 2)
    fmt.Printf("a=%#v\nb=%#v\nc=%#v\n", a, b, c)
}

outputs:

a=[]int{}
b=[]int{4, 3, 2}
c=[]int{4, 3, 2}

So, len(a) == 0, len(b) == 3, len(c) == 3, and the second call to append() owerwrote what the first one did because all the slices share the same underlying array.

As to reallocation of the backing array, the spec is clear:

If the capacity of s is not large enough to fit the additional values, append allocates a new, sufficiently large underlying array that fits both the existing slice elements and the additional values. Otherwise, append re-uses the underlying array.

From this, it follows that:

  1. append() never copies the underlying storage if the capacity of the slice being appeneded to is sufficient.
  2. If there's not enough capacity, the array will be reallocated.

That is, given a slice s to which you want to append N elements, the reallocation won't be done iff cap(s) - len(s) ≥ N.

Hence I suspect your problem is not about unexpected reallocation results but rather about the concept of slices as implemented in Go. The code idea to absorb is that append() returns the resulting slice value, which you're supposed to be using after the call unless you fully understand the repercussions.

I recommend starting with this to fully understand them.

kostix
  • 51,517
  • 14
  • 93
  • 176
  • Thanx for pointing to the blog post. So, in order to achieve the desired (and reproducible) behavior, I could write: `b := make([]int, len(a), len(a)+3); copy(b, a); b = append(b, []int{1, 2, 3})` – kirk Nov 19 '15 at 08:46
  • 1
    @kirk If you're appending to a slice `s`, always *always* write `s = append(s, value)`, so essentially "yes". – Vatine Nov 19 '15 at 10:24
  • @kirk, yes, to make sure appending N elements won't reallocate the backing array, create a slice with enough capacity to hold your current data *plus* N, copy existing slice over to the new one and then append. – kostix Nov 19 '15 at 10:29
0

Thanx for your feedback.

So the solution to gain control of the memory allocation is to do it explicitely (which remembers me that Go is a more a system language than other (scripting) langs):

package main

import (
    "fmt"
)

func main() {

    a1 := make([][]int, 3)
    a2 := make([][]int, 3)
    b := [][]int{{1, 1, 1}, {2, 2, 2}, {3, 3, 3}}
    common1 := make([]int, 0)
    common2 := make([]int, 0, 12) // provide sufficient capacity
    common1 = append(common1, []int{10, 20}...)
    common2 = append(common2, []int{10, 20}...)

    idx := 0
    for _, k := range b {
        a1[idx] = append(common1, k...) // new slice is allocated
        
        a2[idx] = make([]int, len(common2), len(common2)+len(k))
        copy(a2[idx], common2)      // copy & append could probably be
        a2[idx] = append(a2[idx], k...) // combined into a single copy step
        
        idx++
    }

    fmt.Println(a1)
    fmt.Println(a2)
}

output:

[[10 20 1 1 1] [10 20 2 2 2] [10 20 3 3 3]]

[[10 20 1 1 1] [10 20 2 2 2] [10 20 3 3 3]]

https://play.golang.org/p/Id_wSZwb84

Community
  • 1
  • 1
kirk
  • 125
  • 1
  • 5
  • About explicit memory allocation... That's yes and no. A common idiom to append to a slice is to just not bother and do `s = append(s, e1, e2, ...)` -- so that Go will make sure the backing array will be reallocated if appending would require it. This frees you from thinking about memory management *unless* you have other slices pointing to the same data (i.e. you've sliced the original slice before appending to it). But that's a good tradeoff: it makes the simple case simple and the complex cases possible. – kostix Nov 19 '15 at 10:31
  • One more point regarding that idiom I've referred to, is that sometimes you know your slice will have *at least* N element (or may be more) or *at most* N elements. In either case it's possible to do a micro optimization and allocate the slice with length 0 and capacity `N` via `s := make([]T, 0, N)`. In the former case you will be guaranteed that appending the first `N` elements won't reallocate; in the second case you may waste some space trading this for appending speed. In either case, *please don't opmimize before you've profiled!* – kostix Nov 19 '15 at 10:34