0

I am trying to crawl a website using proxy server. Here's my Code

try{
     HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(proxy+link);
     req.Timeout = 60000;//timeout 2 minutes
     req.UserAgent = AGENTLIST[agentCount];
     req.ServicePoint.ConnectionLimit = 1;

                            //response
     HttpWebResponse res = (HttpWebResponse)req.GetResponse();
     Stream dataStream = res.GetResponseStream();
     StreamReader reader = new StreamReader(dataStream);
     string html = reader.ReadToEnd();

     HtmlDocument doc = new HtmlDocument();
     doc.LoadHtml(html);

     body = doc.DocumentNode.SelectSingleNode("//body");
    }catch(Exception e)
    { Console.WriteLine(e.Message);
      Console.WriteLine(e.StackTrace);
      Console.ReadLine();
      }

I get this error

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at ASP.osproxy_getpage_aspx.Page_Load(Object sender, EventArgs e) in c:\SharedCrawl\Dropbox\sharedcrawl\osproxy\GetPage.aspx:line 30

If I do this

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(link); //no proxy 

I do not get error.

Can anybody explain what the error means?

Thanks R

newbieCSharp
  • 181
  • 2
  • 22

1 Answers1

0

That's not how you use proxys on WebRequests. You should use the HttpWebRequest.Proxy property.

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(link);
req.Proxy = new WebProxy(proxyAddress, proxyPort);
markhc
  • 659
  • 6
  • 15