Python requests works fine, when trying same request in golang is not working as expected

Question

Python code, keep in mind that the __cf_bm cookie has to be valid in order to get a json response. The __cf_bm is only valid for around 30min, so you have to get a cookie from https://kick.com on your own to replicate this example.

import requests

headers = {
'Accept':'application/json',
'Cookie':'__cf_bm=yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy',
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}


resp = requests.request('GET', 'https://kick.com/api/v1/channels/xqc', headers=headers)
print (resp.text)

The user-agent is tied to the cookie, make sure to use your user agent with which you got your cookie, not the one in the example. The accept: application/json is not that important, can be omitted.

The problem is when i make the SAME request in golang with the same headers and cookies and user-agent, that i get a cloudflare response, (which is the same response, when the __cf_bm cookie has expired) but i know that the cookie has not expired yet.

Golang Code, go version go1.20.5 linux/amd64

package main

import (
    "compress/gzip"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
)

func MakeRequest(url string) (*string, error) {
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
    req.Header.Set("Accept", "application/json")
    req.Header.Set("Connection", "keep-alive")
    req.Header.Set("Accept-Encoding", "gzip, deflate")
    req.AddCookie(&http.Cookie{Name: "__cf_bm", Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy"})

    client := &http.Client{}
    resp, err := client.Do(req)
    var body string

    if err != nil {
        return &body, err
    }

    defer resp.Body.Close()

    var reader io.ReadCloser
    reader, _ = gzip.NewReader(resp.Body)
    bytes, _ := ioutil.ReadAll(reader)
    body = string(bytes)
    return &body, nil
}

func main() {
    data, _ := MakeRequest("https://kick.com/api/v1/channels/xqc")
    fmt.Println(*data)
}

I added Connection and Accept-Encoding header which python requests adds by default but golang doesn't.

The headers are the same in python and golang but the response is not, i'm doing something wrong or is the way golang sends the request. I can't see what im doing wrong...

It's simpler to let the Go HTTP client handle compressed responses. If you do handle compression in the application (by setting the request accept-encoding header), then the application should examine the content encoding response header and decode as appropriate. Also, handle errors and examine the response status code. — Charlie Tumahai, Jun 20 '23 at 12:56
yes good point, but that does not answer why the same request in python returns json data and in golang does not, how is that even possible? — invictadux, Jun 20 '23 at 13:00
There are no decoding error, the python response status code is 200 with json data and the golang request with same headers, responds with a status code of 403 and html javascript whitch is the cloudflare bot detection script. This same 403 response that i get in golang with a valid __cf_bm cookie, i get in python requests when the __cf_bm cookie expires. I assume that the golang request is messing with the cookie, but that is not possible because i made a server and checked the headers for the golang and python requests and they are exactly the same, and its NOT the decoding of the response. — invictadux, Jun 20 '23 at 16:50

score 1 · Answer 1 · answered Jun 19 '23 at 03:16

1

Is the gzip mod needed?

bytes, _ := ioutil.ReadAll(resp.Body)
body = string(bytes)
return &body, nil

answered Jun 19 '23 at 03:16

corgifarts

11
1

1

yes if you use Accept-Encoding: "gzip, deflate" but the header does not affect the output – invictadux Jun 19 '23 at 20:53
Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jun 20 '23 at 01:16

score -1 · Answer 2 · answered Jun 20 '23 at 21:34

Firstly, the commentator above was right.Gzip mod not needed here. Next the mistake was : if you create a variable of type bytes.Buffer (without initialization) and assign it to a field of type io.Reader, then after checking io.Reader for nil there will be an error: invalid memory address or nil pointer dereference. More about that How to correctly check io.Reader for nil?

The right code:

func main() {
request, err := http.Get("https://kick.com/api/v1/channels/xqc")
if err != nil {
    log.Fatal(err)
}
myCookie := &http.Cookie{
    Name:  "__cf_bm",
    Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy",
}
request.Request.AddCookie(myCookie)
request.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
request.Header.Set("Accept", "application/json")
request.Header.Set("Connection", "keep-alive")

body, err := io.ReadAll(request.Body)
if err != nil {
    fmt.Println(err)
}
defer request.Body.Close()
st := string(body)
fmt.Println(st)
}

`http.Get` returns a response, not a request. The [transport documentation](https://pkg.go.dev/net/http#Transport.DisableCompression) describes why OP's application should handle content encoding. — Charlie Tumahai, Jun 21 '23 at 14:15

Python requests works fine, when trying same request in golang is not working as expected

2 Answers2