Website rejecting HttpWebRequest in Visual Basic

Question

I tried to get the webpage source of pages under www.digikey.com It was working fine long back but now the website is rejecting the request, but it shows no problem in web browser , issue happens when trying to access via code and when i examine the page sorce received now its written

There was a problem with your request. We are unable to process your request. Please return to the previous page to try again or contact Digi-Key Webmaster if you feel that you have received this message in error. Please reference the following incident number so we may assist you with this error.

The code i used was:

    Dim PartURL As String
    PartURL = "http://www.digikey.com"
    Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(PartURL)
    Dim response As System.Net.HttpWebResponse = request.GetResponse()
    Dim sr As System.IO.StreamReader = New System.IO.StreamReader(response.GetResponseStream())
    Dim sourcecode As String = sr.ReadToEnd()

when i changed the url to google.com its working well. The website www.digikey.com works without a problem in web browsers. It shows error only when trying to access via the code. So i thought it may have something in relation with the code. Is it because digikey is rejecting the request. Is there any way i can get the source code of pages under www.digikey.com

this is off-topic for stackoverflow. it doesnt directly use visual-studio-2010 and VBA is a programming language. Your code works and the error is not code related. — , Sep 26 '13 at 10:58
what i meant was is there any way so that the website feels its from a web browser Since the website works well in browser So i thought it may have some relation with the way the website is accessed — 0xAB1E, Sep 26 '13 at 11:03
the website www.digikey.com works without error in web browser It shows only error when tried to access via visual basic — 0xAB1E, Sep 26 '13 at 11:09
See these two answers: http://stackoverflow.com/questions/8278057/httpwebrequest-how-to-identify-as-a-browser http://stackoverflow.com/questions/4568913/httpwebrequest-versus-browser-request — Tony, Sep 26 '13 at 11:26

score 2 · Accepted Answer · answered Sep 26 '13 at 11:28

Try this:

Dim PartURL As String
PartURL = "http://www.digikey.com"
Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(PartURL)
request.UserAgent = "Mozilla/5.0 (Windows NT 5.1; rv:2.0b8) Gecko/20100101 Firefox/4.0b8"
request.Accept = "Accept: text/html,application/xhtml+xml,application/xml"

Dim response As System.Net.HttpWebResponse = request.GetResponse()
Dim sr As System.IO.StreamReader = New System.IO.StreamReader(response.GetResponseStream())
Dim sourcecode As String = sr.ReadToEnd()

Website rejecting HttpWebRequest in Visual Basic

1 Answers1