Go: Passing pointers of an array to gob without copying?

Question

I have a very, very large array (not slice) of maps that I am then trying to encode. I really need to avoid making a copy of the array but I can't figure out how to do this.

So I far I have this:

func doSomething() {
 var mygiantvar [5]map[string]Searcher
 mygiantvar = Load()
 Save(`file.gob.gz`, &mygiantvar)
}

func Save(filename string, variable *[5]map[string]Searcher) error {
    // Open file for writing
    fi, err := os.Create(filename)
    if err !=nil {
        return err
    }
    defer fi.Close()
    // Attach gzip writer
    fz := gzip.NewWriter(fi)
    defer fz.Close()
    // Push from the gob encoder
    encoder := gob.NewEncoder(fz)
    err = encoder.Encode(*variable)
    if err !=nil {
        return err
    }
    return nil
}

From my understanding that will pass a pointer of mygiantvar to Save, which saves the first copy. But then the entire array will surely be copied into encoder.Encode which will then copy it around many more functions, right?

This mygiantvar variable will be something like 10GB in size. So it must avoid being copied ever.

But then again perhaps only the actual array [5] part is copied but the maps inside of this are pointers inside an array, so the array of pointers to maps would be copied instead of the maps themselves? I have no idea about this - it's all very confusing.

Any ideas?

score 2 · Accepted Answer · edited May 23 '17 at 12:09

2

Note that Encoder.Encode will pass around an interface{}.

func (enc *Encoder) Encode(v interface{}) error {

That means a kind of a pointer to whatever you will be passing to it, as I described in "what is the meaning of interface{} in golang?"
(see also "Why can't I assign a *Struct to an *Interface?")

An interface value isn't the value of the concrete struct (as it has a variable size, this wouldn't be possible), but it's a kind of pointer (to be more precise a pointer to the struct and a pointer to the type)

That means it won't copy the full content of your map (or here of your array).
Since array is a value, you could slice it to avoid any copy during the call to Encode():

err = encoder.Encode(*variable[:])

See "Go Slices: usage and internals"

This is also the syntax to create a slice given an array:

x := [3]string{"Лайка", "Белка", "Стрелка"}
s := x[:] // a slice referencing the storage of x

If that doesn't work, you can keep *variable (here an array: [5]map[string]Searcher), as map types are reference types, like pointers or slices: the copy won't be huge.
See "Go maps in action".

While the array will be copied when passed to interface{}, the map content won't be copied.
See this play.golang.org example:

package main

import "fmt"

func main() {
    var a [1]map[string]int
    a[0] = make(map[string]int)
    a[0]["test"] = 0
    modify(a)
    fmt.Println(a)
}

func modify(arr interface{}) {
    a := arr.([1]map[string]int)
    a[0]["test"] = -1
}

Output:

[map[test:-1]]

edited May 23 '17 at 12:09

Community

1
1

answered Aug 03 '14 at 05:58

VonC

1,262,500
529
4,410
5,250

But the array itself is copied that one time where it says `encoder.Encode(*variable)`? – Alasdair Aug 03 '14 at 06:05
@Alasdair yes, but you can slice it to avoid the copy: `*variable[:]` – VonC Aug 03 '14 at 06:08
But if I sliced it then it would be a different type and surely not load back into the struct that defines itself as `[5]map[string]Searcher`? I'm hoping though that even if the array is copied, the maps inside of it are not copied, just a copy of their pointers are made. – Alasdair Aug 03 '14 at 06:16
@Alasdair you are right, and I have edited the answer: map content won't be copied. Only the map itself, which is a reference type. – VonC Aug 03 '14 at 06:20
The interface may contain a pointer, but it's a pointer to freshly-allocated memory that contains the value itself, and that memory is initialized by copying the value over. So, in his array case, it'd allocate space for the size of the array, copy over the contents, and then put a pointer to the new memory in the interface. To demonstrate that interfaces aren't just pass-by-reference, see this [example](http://play.golang.org/p/-ymaQbDimR). – joshlf Aug 03 '14 at 08:24
@joshlf13 what would you recommend then to avoid the copy while keeping the `[5]` array information needed by the Encoder? – VonC Aug 03 '14 at 08:26
@joshlf13 I agree the array will be copied, but it is an array of map, so that should be too bad, should it? – VonC Aug 03 '14 at 08:27
@joshlf13 I mean http://play.golang.org/p/izL7PD0r2a does work: while the array of map might be copied, the map content is still mutable. – VonC Aug 03 '14 at 08:30
Ah, gotcha. I thought you meant that the array itself wouldn't be copied. – joshlf Aug 04 '14 at 04:07

Go: Passing pointers of an array to gob without copying?

1 Answers1