Golang read HTTPS response body

Question

I'm writing some middleware and I need to be able to log the response body content even when the destination is using TLS encryption.

I have a handler chain within which I store the response body in an intermediate buffer, so that I can read it more than once. This is based on the excellent example provided by icza (Golang read request body).

In my handler func, I'm doing this....

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    // Print the response body to stdout
    fmt.Printf("Dest HTTP response body: %s\n", body)

    bRdr := bytes.NewReader(body)
    n, err := io.Copy(w, bRdr) // Copy the entire response body into our outgoing response

What I'm finding is that I get readable output when connection to a destination not using TLS, but when connected to a destination using TLS, it seems the response body is still encrypted, though the Copy into the final response to the originator results in the originator receiving valid response body content.

Is this the expected behaviour for reads of the response body with an encrypted path? Can I decrypt this data to be able make it readable? I've read the http, tls and crypto package documentation, but have not found any clues.

TLS operates at a completely different layer. You are not looking at a TLS stream when you read the body. — JimB, Oct 30 '20 at 14:26
TLS encryption/decryption is transparent at this layer. Are you sure you are looking at the encrypted body, or some garbage data? — Burak Serdar, Oct 30 '20 at 14:28
Thanks all for your comments. Travis seems to have identified the issue I'm having. It appears the response body I'm reading is gzip encoded (the response contains "Content-Encoding: gzip"). In order to verify that this was the case, I had to explicitly remove the "Accept-Encoding: gzip" header that was in the originating request before forwarding and also configure the Transport to set "DisableCompression: true". Once I made both of those changes, I then see responses with no "Content-Encoding" header and the body I read is human readable. — user14549440, Nov 01 '20 at 15:15
Now all I need to do is turn gzip encoding back on and figure out how to unzip the gzipped response body. I see there are various other posts about that topic, so I'll research those. — user14549440, Nov 01 '20 at 15:16

score 2 · Answer 1 · answered Oct 30 '20 at 15:58

I'm not sure if I understand the problem but here is me calling an https google link and printing the output.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"

    "golang.org/x/net/http2"
)

func main() {
    client := &http.Client{Transport: transport2()}

    res, err := client.Get("https://www.google.com")
    if err != nil {
        log.Fatal(err)
    }

    body, err := ioutil.ReadAll(res.Body)
    if err != nil {
        log.Fatal(err)
    }

    res.Body.Close()

    fmt.Printf("Code: %d\n", res.StatusCode)
    fmt.Printf("Body: %s\n", body)
}

func transport2() *http2.Transport {
    return &http2.Transport{
        DisableCompression: true,
        AllowHTTP:          false,
    }
}

score 0 · Answer 2 · answered Nov 09 '20 at 12:09

Thanks all for your comments. Travis seems to have identified the issue I'm having. It appears the response body I'm reading is gzip encoded (the response contains "Content-Encoding: gzip"). In order to verify that this was the case, I had to explicitly remove the "Accept-Encoding: gzip" header that was in the originating request before forwarding and also configure the Transport to set "DisableCompression: true". Once I made both of those changes, I then see responses with no "Content-Encoding" header and the body I read is human readable.

Golang read HTTPS response body

2 Answers2