How do I read a streaming response body using Golang's net/http package?

Question

I am trying to connect to an endpoint that does http streaming of json data. I was wondering how to perform a basic request using Go's net/http package and read the response as it comes in. Currently, I am only able to read the response when the connection closes.

resp, err := http.Get("localhost:8080/stream")
if err != nil {
    ...
}
...
// perform work while connected and getting data

Any insight would be greatly appreciated!

Thanks!

-RC

Did you see [this answer](http://stackoverflow.com/questions/19292113/not-buffered-http-responsewritter-in-golang) and does it help? — Ray Toal, Mar 01 '14 at 00:47
That code will operate on a stream. If you're not getting it when you expect, it's because of how the server is sending it. I've parsed many GBs of responses with this model. — Dustin, Mar 01 '14 at 01:38

score 47 · Answer 1 · edited Oct 24 '18 at 16:45

47

The answer provided by Eve Freeman is the correct way to read json data. For reading any type of data, you can use the method below:

resp, err := http.Get("http://localhost:3000/stream")
...

reader := bufio.NewReader(resp.Body)
for {
    line, err := reader.ReadBytes('\n')
    ...

    log.Println(string(line))
}

edited Oct 24 '18 at 16:45

Eve Freeman

32,467
4
86
101

answered Mar 04 '14 at 16:39

chourobin

4,004
4
35
48

But how does one detect EOF on that []byte? – jsgoecke Apr 12 '16 at 20:52
5

EOF is returned with err – ciscogambo May 10 '16 at 13:53
now using these chunks/lines how can I directly write it to file. Means how write using stream not whole as file. ? – Amulya Kashyap Jul 02 '18 at 13:22

score 17 · Accepted Answer · answered Mar 01 '14 at 14:23

17

The way to do streaming JSON parsing is with a Decoder:

json.NewDecoder(resp.Body).Decode(&yourStuff)

For a streaming API where it's a bunch of objects coming back (a la Twitter), that should stream great with this model and the built-in encoding/json API. But if it's a large response where you have an object that's got a giant array with 10MB of stuff, you probably need to write your own Decoder to pull those inner pieces out and return them. I'm running into that problem with a library I've written.

answered Mar 01 '14 at 14:23

Eve Freeman

32,467
4
86
101

Thanks for the answer! I managed to get it working by creating a bufio.Reader and calling ReadBytes() on the resp.Body. – chourobin Mar 04 '14 at 16:36
Eve, where documentation about max 10MB? i need information – KingRider Dec 23 '16 at 17:35
It's not a max, it's just the way the parser works, objects at a time. – Eve Freeman Dec 27 '16 at 00:45
Is there a way to listen for when the response has finished being written? – Alexander Mills Oct 24 '18 at 00:25

How do I read a streaming response body using Golang's net/http package?

2 Answers2