Convert between slices of different types

Question

I get a byte slice ([]byte) from a UDP socket and want to treat it as an integer slice ([]int32) without changing the underlying array, and vice versa. In C(++) I would just cast between pointer types; how would I do this in Go?

Answer 1

As others have said, casting the pointer is considered bad form in Go. Here are examples of the proper Go way and the equivalent of the C array casting.

WARNING: all code untested.

The Right Way

In this example, we are using the encoding/binary package to convert each set of 4 bytes into an int32. This is better because we are specifying the endianness. We are also not using the unsafe package to break the type system.

import "encoding/binary"

const SIZEOF_INT32 = 4 // bytes

data := make([]int32, len(raw)/SIZEOF_INT32)
for i := range data {
    // assuming little endian
    data[i] = int32(binary.LittleEndian.Uint32(raw[i*SIZEOF_INT32:(i+1)*SIZEOF_INT32]))
}

The Wrong Way (C array casting)

In this example, we are telling Go to ignore the type system. This is not a good idea because it may fail in another implementation of Go. It is assuming things not in the language specification. However, this one does not do a full copy. This code uses unsafe to access the "SliceHeader" which is common in all slices. The header contains a pointer to the data (C array), the length, and the capacity. Instead of just converting the header to the new slice type, we first need to change the length and capacity since there are less elements if we treat the bytes as a new type.

import (
    "reflect"
    "unsafe"
)

const SIZEOF_INT32 = 4 // bytes

// Get the slice header
header := *(*reflect.SliceHeader)(unsafe.Pointer(&raw))

// The length and capacity of the slice are different.
header.Len /= SIZEOF_INT32
header.Cap /= SIZEOF_INT32

// Convert slice header to an []int32
data := *(*[]int32)(unsafe.Pointer(&header))

Answer 2

You do what you do in C, with one exception - Go does not allow to convert from one pointer type to another. Well, it does, but you must use unsafe.Pointer to tell compiler that you are aware that all rules are broken and you know what you are doing. Here is an example:

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    b := []byte{1, 0, 0, 0, 2, 0, 0, 0}

    // step by step
    pb := &b[0]         // to pointer to the first byte of b
    up := unsafe.Pointer(pb)    // to *special* unsafe.Pointer, it can be converted to any pointer
    pi := (*[2]uint32)(up)      // to pointer to the first uint32 of array of 2 uint32s
    i := (*pi)[:]           // creates slice to our array of 2 uint32s (optional step)
    fmt.Printf("b=%v i=%v\n", b, i)

    // all in one go
    p := (*[2]uint32)(unsafe.Pointer(&b[0]))
    fmt.Printf("b=%v p=%v\n", b, p)
}

Obviously, you should be careful about using "unsafe" package, because Go compiler is not holding your hand anymore - for example, you could write pi := (*[3]uint32)(up) here and compiler wouldn't complain, but you would be in trouble.

Also, as other people pointed already, bytes of uint32 might be layout differently on different computers, so you should not assume these are layout as you need them to be.

So safest approach would be to read your array of bytes one by one and make whatever you need out of them.

Alex

Answer 3

The short answer is you can't. Go wont let you cast a slice of one type to a slice of another type. You will have loop through the array and create another array of the type you want while casting each item in the array. This is generally regarded as a good thing since typesafety is an important feature of go.

Answer 4

Since Go 1.17, there is a simpler way to do this using the unsafe package.

import (
    "unsafe"
)

const SIZEOF_INT32 = unsafe.Sizeof(int32(0)) // 4 bytes

func main() {
    var bs []byte
    
    // Do stuff with `bs`. Maybe do some checks ensuring that len(bs) % SIZEOF_INT32 == 0
    
    data := unsafe.Slice((*int32)(unsafe.Pointer(&bs[0])), len(bs)/SIZEOF_INT32)

    // A more verbose alternative requiring `import "reflect"`
    // data := unsafe.Slice((*int32)(unsafe.Pointer((*reflect.SliceHeader)(unsafe.Pointer(&bs)).Data)), len(bs)/SIZEOF_INT32)
}

Answer 5

Go 1.17 and beyond

Go 1.17 introduced the unsafe.Slice function, which does exactly this.

Converting a []byte to a []int32:

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    theBytes := []byte{
        0x33, 0x44, 0x55, 0x66,
        0x11, 0x22, 0x33, 0x44,
        0x77, 0x66, 0x55, 0x44,
    }

    numInts := uintptr(len(theBytes)) * unsafe.Sizeof(theBytes[0]) / unsafe.Sizeof(int32(0))
    theInts := unsafe.Slice((*int32)(unsafe.Pointer(&theBytes[0])), numInts)

    for _, n := range theInts {
        fmt.Printf("%04x\n", n)
    }
}

Playground.

Answer 6

I had the size unknown problem and tweaked the previous unsafe method with the following code. given a byte slice b ...

int32 slice is (*(*[]int)(Pointer(&b)))[:len(b)/4]

The array to slice example may be given a fictional large constant and the slice bounds used in the same way since no array is allocated.

Answer 7

You can do it with the "unsafe" package

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    var b [8]byte = [8]byte{1, 2, 3, 4, 5, 6, 7, 8}
    var s *[4]uint16 = (*[4]uint16)(unsafe.Pointer(&b))
    var i *[2]uint32 = (*[2]uint32)(unsafe.Pointer(&b))
    var l *uint64 = (*uint64)(unsafe.Pointer(&b))

    fmt.Println(b)
    fmt.Printf("%04x, %04x, %04x, %04x\n", s[0], s[1], s[2], s[3])
    fmt.Printf("%08x, %08x\n", i[0], i[1])
    fmt.Printf("%016x\n", *l)
}

/*
 * example run:
 * $ go run /tmp/test.go
 * [1 2 3 4 5 6 7 8]
 * 0201, 0403, 0605, 0807
 * 04030201, 08070605
 * 0807060504030201
 */

Answer 8

Perhaps it was not available when the earlier answers were given, but it would seem that the binary.Read method would be a better answer than "the right way" given above.

This method allows you to read binary data from a reader directly into the value or buffer of your desired type. You can do this by creating a reader over your byte array buffer. Or, if you have control of the code that is giving you the byte array, you can replace it to read directly into your buffer without the need for the interim byte array.

See https://golang.org/pkg/encoding/binary/#Read for the documentation and a nice little example.

Answer 9

http://play.golang.org/p/w1m5Cs-ecz

package main

import (
    "fmt"
    "strings"
)

func main() {
    s := []interface{}{"foo", "bar", "baz"}
    b := make([]string, len(s))
    for i, v := range s {
        b[i] = v.(string)
    }
    fmt.Println(strings.Join(b, ", "))
}

Answer 10

func crackU32s2Bytes(us []uint32) []byte {
    var bs []byte
    var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
    var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
    ptrBs.Data = ptrUs.Data
    ptrBs.Len = ptrUs.Len*4
    ptrBs.Cap = ptrBs.Len
    return bs
}

func crackBytes2U32s(bs []byte) []uint32 {
    var us []uint32
    var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
    var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
    ptrUs.Data = ptrBs.Data
    ptrUs.Len = ptrBs.Len/4
    ptrUs.Cap = ptrUs.Len
    return us
}