.net downloadstring(url) returning 503 error suddenly

Question

I have a simple function to scrape a web page that is suddenly getting a 503 error from the following code for a specific page:

try
{
  WebClient client = new WebClient();
  client.Proxy = null;
  string webhtml = client.DownloadString(strURL);
  client.Dispose();
  return (webhtml);
}

The url works when I enter it in IE or FF but it's returning an error when this code runs. Other urls work fine. I'm guessing there is something different about the http request from .net than from either of the browsers. Is there any way to better emulate a browser-based http request?

Thanks.

They could be banning your request because they don't want their site to be scraped. — idstam, Jun 28 '11 at 04:25
Is it possible to post the URL? I could help more if I had that. — cgatian, Jul 01 '11 at 08:22

score 0 · Accepted Answer · answered Jun 28 '11 at 05:26

0

Found the http headers I was looking for from this site. Very useful.

http://www.ericgiguere.com/tools/http-header-viewer.html

answered Jun 28 '11 at 05:26

nycdan

2,819
2
21
33

Try [Fiddler](http://www.fiddler2.com/fiddler2/) out. It has all the goodness you'll ever need. – cgatian Jul 01 '11 at 08:21
Thanks, cgatian. I forgot about Fiddler but you're right. That would have solved my problem just as well. – nycdan Jul 16 '11 at 02:30

.net downloadstring(url) returning 503 error suddenly

1 Answers1