I am using a vb.net webclient and am having difficulty connecting to certain sites and retrieving content. It seems to be that certain sites recognise this is a webclient and not a real browser.
I am using a cookie aware version of the webclient, but the sites in question come back with a 500 error, i'm certain they are being blocked by the web servers as using a normal browser to the same URL will work fine.
I've added user-agent headers and also tried a few others (cache policy, pragma, encoding) but none seem to help.
Does anybody know of a way to make the webclient act more like a real browser?
Thanks