Calculating sha256 gives different results after appending slices depending on if I print out the slice before or not

Question

I am calculating a sha256 from multiple strings. I convert them to byte slices in a specific way and append them all together and then compute the hash using the built in library. However, depending on if I print out the slice before calculating the sha256 or not I wierdly get different results. When testing it in playground I cannot reproduce it.

The tested code can been seen and run on https://play.golang.org/p/z8XKx-p9huG where it actually gives the same result in both cases.

func getHash(input1 string, input2hex string, input3hex string, input4 string) (string, error) {
    input1bytes := []byte(input1)
    firstHalfinput1Bytes := input1bytes[:8]
    secondHalfinput1Bytes := input1bytes[8:16]

    input4Bytes := []byte(input4)

    input3Bytes, err := hex.DecodeString(input3hex)
    if err != nil {
        fmt.Println("err " + err.Error())
    }

    input2Bytes, err := hex.DecodeString(input2hex)
    if err != nil {
        fmt.Println("err " + err.Error())
    }

    fullHashInputBytes := append(firstHalfinput1Bytes, input2Bytes...)
    // THIS IS THE OPTIONAL PRINT WHICH CHANGES OUTPUT LOCALLY:
    fmt.Println("fullHashInputBytes", fullHashInputBytes)
    fullHashInputBytes = append(fullHashInputBytes, secondHalfinput1Bytes...)
    fullHashInputBytes = append(fullHashInputBytes, input3Bytes...)
    fullHashInputBytes = append(fullHashInputBytes, input4Bytes...)
    sha256 := sha256.Sum256(fullHashInputBytes)
    for i := 0; i < 8; i++ {
        sha256[i+16] ^= sha256[i+24]
    }
    hash := hex.EncodeToString(sha256[:24])
    fmt.Println("hash", hash)
    return hash, nil
}

The logs on the playground are

Hello, playground
fullHashInputBytes [84 72 73 83 73 83 78 79 30 0 22 121 57 203 102 148 210 196 34 172 210 8 160 7]
hash 0161d9de8dd815ca9f4e1c7bb8684562542cc24b1026321c
hash 0161d9de8dd815ca9f4e1c7bb8684562542cc24b1026321c

but if I run EXACTLY the same code locally (just copy-paste it into a main.go and do go run main.go or go build . and ./test) I get

Hello, playground
fullHashInputBytes [84 72 73 83 73 83 78 79 30 0 22 121 57 203 102 148 210 196 34 172 210 8 160 7]
hash 0161d9de8dd815ca9f4e1c7bb8684562542cc24b1026321c
hash d2de4ffb4e8790b8fd1ceeba726436fd97875a5740c27b47

I'm using go version 1.13.4 but had the same issue with 1.10.4. I also get the same issue on both my local machine and when deployed to our server.

Answer 1

This is because you create fullHashInputBytes by appending to firstHalfinput1Bytes first:

fullHashInputBytes := append(firstHalfinput1Bytes, input2Bytes...)

Which is a slice of the input1bytes:

firstHalfinput1Bytes := input1bytes[:8]

So the first append may overwrite the contents of input1bytes at indices higher than 7, which is actually the content of secondHalfinput1Bytes:

secondHalfinput1Bytes := input1bytes[8:16]

So later when you also append secondHalfinput1Bytes to fullHashInputBytes, you might end up appending different contents.

This is most likely not what you want.

If you do it "clean":

var fullHashInputBytes []byte
fullHashInputBytes = append(fullHashInputBytes, firstHalfinput1Bytes...)
fullHashInputBytes = append(fullHashInputBytes, input2Bytes...)
// OPTIONAL print doesn't change anything:
fmt.Println("fullHashInputBytes", fullHashInputBytes)
// ...rest of your appends...

Then output will be the same if you run it locally or on the Go Playground.

Why the deviant behavior?

Whether your first append overwrites input1bytes depends on whether the append can be performed "in-place", without having to allocate a new backing array, which depends on the capacity of firstHalfinput1Bytes, which is "inherited" from input1bytes:

input1bytes := []byte(input1)
fmt.Println(cap(input1bytes))

(You may read about it in greater detail here: Concatenate two slices in Go).

The conversion []byte(input) only guarantees to have the bytes of input1, but the spec does not dictate how big the capacity of the resulting slice should be. And it may depend on platform / architecture. On the Go Playground the above conversion results in capacity = 16, on my local amd64 architecture I get capacity = 32.

One final piece: the capacity used for the slice of the result of the []byte(input) conversion may depend on what you do with the result slice. The compiler may make decisions to use lower capacity if you pass it to fmt.Println() as this signals that the slice may escape. Again, the decision made by the compiler is out of your hands.

Don't rely on such thing, do not write code that relies on the capacity of the resulting slice of a conversion. Do it in the "clean" way: do not append to firstHalfinput1Bytes but to a new slice.