Golang: Get request sends me html contents but works fine from python and curl

Question

I am trying to call a simple api by using golang. But, each time it sends me html content of login page instead of actual data. But same get request works from python and curl.

func main() {
    client := &http.Client{}
    req, err := http.NewRequest("GET", "https://www.lrn.com", nil)
    if err != nil {
        os.Exit(1)
    }

    q := req.URL.Query()
    q.Add("phoneList", "XXXXXX")
    q.Add("output", "json")
    q.Add("version", "5")
    //req.URL.RawQuery = q.Encode()
    req.Header.Set("loginId", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

    fmt.Println(req.URL.String())

    resp, err := client.Do(req)

    if err != nil {
        fmt.Println("Errored when sending request to the server")
        return
    }

    defer resp.Body.Close()
    resp_body, _ := ioutil.ReadAll(resp.Body)

    fmt.Println(resp.Status)
    fmt.Println(string(resp_body))
}

Above script gives me html content of login page. But if i use python, it works just fine.

import requests

r=requests.get("https://www.lrn.com", params = {'version':'5', "phoneList":"XXXXXX", "output":"json"}, headers={"loginId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"})

print r.text

Could someone please explain me what might be wrong in my golang script.

Use a debugging proxy like Fiddler or Charles to see how the two requests differ. Perhaps a header or cookie is missing? — Panagiotis Kanavos, Sep 19 '20 at 19:34
@SteffenUllrich sorry even uncommenting those param results same. — ryan embgrets, Sep 19 '20 at 20:35
@PanagiotisKanavos thanks, i do think that i am missing some header or cookie in request but due to https i cannot see much. Let me try fiddle and see if that sheds some light. — ryan embgrets, Sep 19 '20 at 20:37
One other thing to try: both `curl` and `python` will send a `User-Agent` header e.g. `python-requests/2.24.0`. Try mimicking a user-agent to see if that makes a difference. — colm.anseo, Sep 19 '20 at 21:03
@colm.anseo i think my `loginId` header does not reach api server. If i remove it from postman file i get same login page html in postman response like i do in golang. Might be due to some redirect it does not reach the api server? — ryan embgrets, Sep 19 '20 at 21:33

Steffen Ullrich · Accepted Answer · 2020-09-20T07:18:54.300

1

//req.URL.RawQuery = q.Encode()
req.Header.Set("loginId", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

fmt.Println(req.URL.String())

The last line outputs the URL it uses. With the code above this will be https://www.lrn.com. Given that you want to have query parameters the expected value would be instead https://www.lrn.com?output=json&... though. So something is clearly missing from your construction of the URL.

When activating the line you've explicitly commented out the result is very different:

req.URL.RawQuery = q.Encode()  // activate this line
...    
fmt.Println(req.URL.String())

In this case I get https://www.lrn.com?output=json&phoneList=XXXXXX&version=5, which is the expected value. And with this value the result from the server is for me the same as for your Python code.

But this seems to be only the first problem. Based on input in the comments I could track down the second problem, which was caused by a broken server together with header normalization by Golang. Based on the comments I've now used a different URL https://www.dncscrub.com/app/main/rpc/scrub where the Python code worked and the Golang code again did not work.

By comparing the HTTP requests used by both and reducing to their essential difference it showed, that the server erroneously interpreted a HTTP request header in a case-sensitive way. This worked

GET /app/main/rpc/scrub?version=5&phoneList=2123727200 HTTP/1.1
Host: www.dncscrub.com
loginId: 0610A62F

And resulted in the expected HTTP/1.1 401 Invalid credentials. While this did not work

GET /app/main/rpc/scrub?version=5&phoneList=2123727200 HTTP/1.1
Host: www.dncscrub.com
Loginid: 0610A62F

but resulted in a redirect HTTP/1.1 302 Found.

The only difference between these requests is the use of loginId: ... vs. Loginid: .... The server is clearly broken by treating HTTP header fields as case-sensitive, see Are HTTP headers case-sensitive? which also links to the relevant standards.

Since req.Header.Set will normalize the header field one cannot use it. A work around is to manipulate the dictionary instead, which is usually not a good idea:

req.Header["loginId"] = []string{"0610A62F"}

See also GoLang: Case-sensitive HTTP Headers with net/http for a similar problem.

edited Sep 20 '20 at 07:18

answered Sep 19 '20 at 20:19

Steffen Ullrich

114,247
10
131
172

I have activated the line `req.URL.RawQuery = q.Encode()` but i still get html content through golang. :( – ryan embgrets Sep 19 '20 at 20:59
@ryanembgrets: All I can say is that I get exactly the same answer as with your Python code - both being HTML. It might be that something will be different if actual values are provided where you currently show 'XXXX....'. But I have no access to these values so I cannot reproduce and this way debug your problem. Note though that `www.lrn.com` redirects to `lrn.com` so you might try to use this in the direct URL instead and check if it gets better. – Steffen Ullrich Sep 19 '20 at 21:06
Sorry, i have provided dummy values for the code snippet. To, me it looks like application cannot find 'loginId' header in request which contains authentication due to which it gives login html contents. – ryan embgrets Sep 19 '20 at 21:25
If i remove the `loginId` header from the postman i get the same html content in postman like i do in the case of golang. So, i think while using golang my header does not reach api server. Could it be due to some redirect where this header does not gets forwarded? – ryan embgrets Sep 19 '20 at 21:30
@ryanembgrets: As I suggested earlier: try accessing `lrn.com` instead of `www.lrn.com` so that you don't get redirected. – Steffen Ullrich Sep 20 '20 at 04:58
@ryanembgrets: based on [this report](https://github.com/golang/go/issues/20042) you need at least golang version 1.8 so that the header is copied on redirect. It is not clear which version you are using. – Steffen Ullrich Sep 20 '20 at 05:04
https://play.golang.org/p/KrnxWLVj8s2 Here you would have the actual data except loginId header. You would see tha golang still sends the html content. But if i use python. import requests r = requests.get("https://www.dncscrub.com/app/main/rpc/scrub", params={'version': '5', "phoneList": "2123727200"}, headers={"loginId": "0610A62F"}) print (r.text) I get the invalid api credentails. Which makes sense due to incorrect header. I really appreciate your time for answering my questions. – ryan embgrets Sep 20 '20 at 06:20
@ryanembgrets: See updated answer. In short: the server is broken by interpreting HTTP headers in a case-sensitive way. – Steffen Ullrich Sep 20 '20 at 07:13
I cannot thank you enough. I have been banging my head against the wall from last 2 days. Happily accepted the answer :) Thank you again. – ryan embgrets Sep 20 '20 at 08:45

Golang: Get request sends me html contents but works fine from python and curl

1 Answers1