I am trying to crawl a website using proxy server. Here's my Code
try{
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(proxy+link);
req.Timeout = 60000;//timeout 2 minutes
req.UserAgent = AGENTLIST[agentCount];
req.ServicePoint.ConnectionLimit = 1;
//response
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
Stream dataStream = res.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string html = reader.ReadToEnd();
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(html);
body = doc.DocumentNode.SelectSingleNode("//body");
}catch(Exception e)
{ Console.WriteLine(e.Message);
Console.WriteLine(e.StackTrace);
Console.ReadLine();
}
I get this error
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at ASP.osproxy_getpage_aspx.Page_Load(Object sender, EventArgs e) in c:\SharedCrawl\Dropbox\sharedcrawl\osproxy\GetPage.aspx:line 30
If I do this
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(link); //no proxy
I do not get error.
Can anybody explain what the error means?
Thanks R