I am developing a web crawler in asp.net.
My web-crawler has a main page from which it opens other HTML pages from HREF tags, downloads the HTML page and fetches the data.
The problem now is sometimes I get an exception of object reference not found sometimes the code works perfectly.
Is it because the asp.net buffer memory is full? If you want I can show you my code.