Integer overflow - golang

Question

I am experiencing some integer overflow error. I have a micro service application built using golang and go-micro as micro service framework. And I am using NATS as message broker.

my micro service payloads are in the format

 map[string]interface{}

The problem occurs when I publish a payload which contains a uint64 such as

 var id uint64 = 512281913614499841

 message := map[string]inteface{}
 message["id"] = id

(this is a unique ID generated by cockroachdb), when a subscriber receives this message as byte and unmarshals this to a uint64, i realize that a overflow occurs and the value becomes

512281913614499840 //this should be 512281913614499841

notice the 0 at the end instead of 1

I have created 2 functions (overFlowError and noOverflow) - see below.

the function overFlowError simulates code that causes overflow

and noOverflow prints the correct result because i use payload in this format map[string][struct] instead of map[string]interface

type UserType struct {
    Email string `json:"email"`
    ID    int64  `json:"id"`
}
func overFlowError() {

        var id int64 = 512281913614499841

        user := UserType{
            Email: "example",
            ID:    id,
        }

        message := map[string]interface{}{
            "data": user,
        }

        //mashal to byte to simulate service payload
        servicePayload, err := json.Marshal(message)

        if err != nil {
            log.Println(err)
        }

        var receivedMessage map[string]interface{}

        json.Unmarshal(servicePayload, &receivedMessage)

        var myUser UserType

        mapstructure.Decode(receivedMessage["data"], &myUser)

        log.Println("---receivedMessage:", myUser.ID) //prints 512281913614499840 - incorrect
    }

No over flow

type UserType struct {
    Email string `json:"email"`
    ID    int64  `json:"id"`
}
func noOverflow() {

        var id int64 = 512281913614499841

        message := map[string]UserType{}

        message["data"] = UserType{
            Email: "example",
            ID:    id,
        }

        byteMessage, err := json.Marshal(message)

        if err != nil {
            log.Println(err)
        }

        var msgMap map[string]UserType

        json.Unmarshal(byteMessage, &msgMap)

        log.Println("---myUser:", msgMap["data"].ID) // prints 512281913614499841 - correct
    }

to over avoid a lot of code rewrite, I am left with the first option which i have simulated in the overFlowError function and also explained above

is there a fix to this problem?

can you please add the code from your repo to this question? — Yaron Schwimmer, Dec 15 '19 at 16:02
This is because the value of `id` overflowing `float64`'s range of precision. You can marshal it as a string and parse it when receiving, or use `json.Decoder` with `json.Decoder.UseNumber` instead of `json.Unmarshal` and then parse the number. — leaf bebop, Dec 15 '19 at 16:05
Thanks leaf, i tried the first solution earlier, however it did not feel like the best solution because a string is not the intended type. I am searching for a more natural and elegant solution. For the second solution, I am thinking of how that would work for a uint64 field in a struct such as this : type User struct{ID uint64} as this also overflows when deserializing. ill have a look at the json.decoder package — Neo, Dec 15 '19 at 19:15
@ykel what numeric properties are you looking from the `ID` field so that you don't want it to become a string? Do you add, subtract, divide or multiply it? — zerkms, Dec 15 '19 at 19:56
@zerkms, if nothing else better works, i'll go with the first solution from leaf. — Neo, Dec 15 '19 at 21:02

torek · Answer 1 · 2019-12-15T23:52:39.800

This is not an integer overflow. As leaf bebop noted in a comment, the problem is actually that JSON numbers tend to get represented as floating-point, and float32 has too few bits to hold the correct value. Even float64, which Go uses by default, is not quite adequate.

See all the answers to JSON integers: limit on size (not just the accepted one), and Tom Christie's comment here. Or, see the older question JSON Not converting long numbers appropriately with its accepted answer.

Depending on what kind of interoperability you need, you can continue to decode these as numbers but use a trickier decoder and/or custom unmarshaler, or just resort to having it encoded in JSON as a string. In practice, when using JSON across Go + Python + TypeScript + various C++ libraries, we¹ have resorted to string encoding. We found this to be far more reliable in the end. (We have not had to do this for 32-bit integers, just for the 64-bit ones.)

To use a non-default decoder, see in particular json.NewDecoder and its UseNumber option.

¹"We" here is not the royal "we", but rather $job, where I have touched all of these except for TypeScript.

Integer overflow - golang

1 Answers1