0

I am trying to send a GET request to download the HTML content of http://footlocker.com/ :

Console.WriteLine(new WebClient().DownloadString("http://footlocker.com"));

But I get a 403 error. To test, I use Python to try and send a GET request (requests library) and I successfully received a 200 response as well as the HTML content:

r = requests.get('http://footlocker.com')
print(r.text)

To see the difference, I printed the headers in the Python request and this is what I got:

{'User-Agent': 'python-requests/2.13.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

So I tried to send the WebClient request with the Python requests User-Agent string:

WebClient wc = new WebClient();
wc.Headers[HttpRequestHeader.UserAgent] = "python-requests/2.13.0";
Console.WriteLine(wc.DownloadString("http://footlocker.com"));

But I still got a 403. What may be the difference between Python's request library and WebClient? Am I missing something obvious here? Why is this happening?

RaghavJhavar
  • 29
  • 1
  • 5
  • if the issue is with the header maybe this will help `wc.Headers.Add("User-Agent : python-requests/2.13.0");`, basically you can try replicating the headers put in by python – thunderbird Jul 16 '17 at 14:09
  • It gives an error saying: `Specified value has invalid HTTP Header characters.` So I changed it to `wc.Headers.Add("User-Agent", "python-requests/2.13.0");` and I get a 403. – RaghavJhavar Jul 16 '17 at 14:11

1 Answers1

1

Figured it out, needed to add this header:

wc.Headers.Add("Accept", "*/*");

Final code:

WebClient wc = new WebClient();
wc.Headers.Add("User-Agent", "python-requests/2.13.0");
wc.Headers.Add("Accept", "*/*");
Console.WriteLine(wc.DownloadString("http://footlocker.com"));
RaghavJhavar
  • 29
  • 1
  • 5
  • It would be better if you use [HttpClient](https://stackoverflow.com/questions/20530152/need-help-deciding-between-httpclient-and-webclient) – Givi Jul 16 '17 at 14:42