Golang and large memory chunk allocation

Question

I have the rather stupid benchmark to test the efficiency of memory allocation:

package main

import (
    "time"
    "fmt"
)

func main() {

    size := 1024 * 1024 * 1024

    t := time.Now()
    for j := 0; j < 100; j += 1 {
        a := make([]int, size)
        for i := 0; i < size; i += 1 {
            a[i] = i
        }
        a = nil
    }
    t1 := time.Now()
    fmt.Printf("Duration: %1d", t1.Sub(t).Seconds())
}

It takes roughly 2-3 minutes on Mac Pro with 16GB of RAM, the memory usage is consistent at 5-8 GB for the process.

The very similar code in Java takes 3 GB of memory and completes in 30 seconds.

What am I missing here?

"What am I missing here?" Nothing. Your code does what it does and it takes as long as it needs. That's all. There is no need to think about artificial microbenchmarks which mix up allocation, GC, CPU speed and RAM speed. — Volker, Jun 01 '17 at 05:30

icza · Accepted Answer · 2017-06-01T09:24:22.203

What am I missing here?

In Java, size of the int type is fixed 4 bytes. In Go int is an architecture dependent type, on 32-bit architectures it's 32 bits (4 bytes), and on 64-bit architectures it's 64 bits (8 bytes).

Most likely you're running it on a 64-bit arch. Which means the size of the Go slice / array you allocate is 8 * 1 GB = 8 GB, while in Java it's only 4 * 1 GB = 4 GB.

Moreover, since you're using int in your loop, Java only has to increment and set 4-byte values, while in Go you're incrementing and setting 8-byte values (type of i and j will be int).

Change your Go code to use int32, and give it a go again.

Also note that your memory usage measurement is flawed, as just the array size in Java is 4 GB and 8 GB in Go, so that 3 GB for Java and 5-8 GB in Go is not the total memory usage!

Also note that []int in Go is a slice and not an array, they are not the same. Slices in Go are struct-like headers containing a pointer to a backing array (see reflect.SliceHeader for details), so there is an implicit indirection step involved using them. For details see Why have arrays in Go? Also related: Array vs Slice: accessing speed

One last note: your code does not measure memory allocation, as that is just the tiny part of the app's execution time. The majority (like 99.99999%) of the execution time is to increment the loop variable a billion times and to fill the arrays with a billion elements.

Golang and large memory chunk allocation

1 Answers1