56

I have a very simple Go webserver. It's job is to receive an inbound json payload. It then publishes the payload to one or more services that expect a byte array. The payload doesn't need to be checked. Just sent over.

In this case, it receives an inbound job and sends it to Google PubSub. It might be another service - it doesn't really matter. I'm trying to find the most efficient way to convert the object to a byte array without first decoding it.

Why? Seems a bit wasteful to decode and convert to JSON on one server, only to unmarshal it later. Plus, I don't want to maintain two identical structs in two packages.

How is it possible to convert the io.ReadCloser to a byte array so I only need to unmarshal once. I tried something like this answer but don't think that's the most efficient way either:

From io.Reader to string in Go

My http server code looks like this:

func Collect(d DbManager) http.HandlerFunc {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json; charset=utf-8")

    code := 422
    obj := Report{}
    response := Response{}
    response.Message = "Invalid request"

    decoder := json.NewDecoder(r.Body)
    decoder.Decode(&obj)

    if obj.Device.MachineType != "" {
        msg,_ := json.Marshal(obj)
        if d.Publish(msg, *Topic) {
          code = 200
        }
        response.Message = "Ok"
    }

    a, _ := json.Marshal(response)
    w.WriteHeader(code)
    w.Write(a)
    return
})
}
Community
  • 1
  • 1
Jenny Blunt
  • 1,576
  • 1
  • 18
  • 41

1 Answers1

88

You convert a Reader to bytes, by reading it. There's not really a more efficient way to do it.

body, err := io.ReadAll(r.Body)

If you are unconditionally transferring bytes from an io.Reader to an io.Writer, you can just use io.Copy

JimB
  • 104,193
  • 13
  • 262
  • 255
  • Cool, I'd tried that but it was outputting null... I hadn't realised that trying to do so after a decode would do this. Two lessons learned, cheers big ears. – Jenny Blunt Oct 09 '16 at 18:17
  • It's a good idea to keep in mind that `io.Copy` makes an allocation. You'll probably want to use `io.CopyBuffer` in the hot path. – Ammar Bandukwala Oct 09 '16 at 20:44
  • 2
    @AmmarBandukwala, unless you have a buffer already, they're identical, Copy just calls CoffyBuffer. You need to make the allocation at some point, and in the scheme of a std lib http request, it's one of many such allocations and is likely to be insignificant. – JimB Oct 09 '16 at 21:04