EOF error after writing HTTP response body to file

Question

I'd like to save a JSON response to a text file before parsing it:

req, err := http.NewRequest("POST", url, body)
req.Header.Set("Authorization", "secret_key")
req.Header.Set("Content-Type", "application/json")

resp, err := client.Do(req)
if err != nil {
    log.Fatal(err)
}
defer resp.Body.Close()

f, err := os.Create("./response.json")
if err != nil {
    log.Fatal(err)
}
defer f.Close()
io.Copy(f, resp.Body)

var result JSONResult
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
    log.Fatal(err)
}

It successfully writes the JSON to a file but then fails on the decoding step with an error that just says EOF. If I parse before writing to file it parses ok, but then the file is empty. Can someone please explain what's happening here? Thanks!

zerkms · Accepted Answer · 2018-08-23T00:42:12.270

3

http.Response.Body is of type io.ReadCloser, which can only be read once (as you can see it does not have a method to rewind).

So alternatively for decoding purposes you could read your just created file.

Or if the response is not large (or you could trim it with io.LimitReader) - you can read it into a buffer

(not tested, something along these lines):

f, err := os.Create("./response.json")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

var buf bytes.Buffer
tee := io.TeeReader(r.Body, &buf)

io.Copy(f, tee)

var result JSONResult
if err := json.NewDecoder(buf).Decode(&result); err != nil {
    log.Fatal(err)
}

References:

https://golang.org/pkg/io/#TeeReader

edited Aug 23 '18 at 00:42

answered Aug 23 '18 at 00:35

zerkms

249,484
69
436
539

Great, thanks! I just started learning Go and I don't see anything in the docs that explains that io.ReadCloser can be read only once. Where can I read more about it? – Vincent Aug 23 '18 at 00:42
1

@Vincent to be honest I cannot find where it's documented, but I think it's implied: if a stream does not have a way to rewind - it's read-once. – zerkms Aug 23 '18 at 00:45
1

@Vincent have a look at https://golang.org/pkg/io/#Seeker If something can be rewinded - then it implements `io.Seeker` (and `r.Body` does not) – zerkms Aug 23 '18 at 00:47
@Vincent: That you cannot read from a io.Reader twice is obvious if you think about what Read does: Read reads some part of a stream return data until EOF. If Reads method does not state something along "After returning EOF further calls to Read will re-read the stream." you cannot assume that Read does this. – Volker Aug 23 '18 at 03:59
@Volker Maybe it's obvious to you, but not to someone who started learning Go 5 days ago, after 5 years of using Python. There are no such concepts in Python. Thanks for downvoting my question! – Vincent Aug 23 '18 at 04:16
@Vincent get +1 back to recover it :-) – zerkms Aug 23 '18 at 04:19
1

@zerkms Spasibo! :) – Vincent Aug 23 '18 at 04:26
@Vincent Can you magically reconsume an iterator in Python? Or does reading a file in Python magically start over once your reach the end? For the downvote: I'm just too lazy too look up the duplicate. – Volker Aug 23 '18 at 05:27
1

@Volker I tend to think that for every Go question here on SO there is a good icza's answer :-D – zerkms Aug 23 '18 at 05:54

EOF error after writing HTTP response body to file

1 Answers1