-1

I am trying in C# to screen scrap two airlines site so I can compare the two fares over many different dates. I manage to do on qua.com but when I try to do it on amadeus.net, I encounter that this site give me a response of

older browser not supported

So using webbrowser class doesn't work... using httpwebrequest doesnt work also. So I want to use webclient but because amadeus.net is heavily base on js or something. I do not know where to post url.

Any suggestion?

Edit: webclient.downloadstring also doesn't wort

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
Photonic
  • 1,316
  • 19
  • 31
  • 1
    Amadeus' website seems to be [using JavaScript to call their own api](http://i.imgur.com/c8wpv2T.png). Find out how to get your access token the same way their website does and you'll have a better solution than scrapping off the info out of their HTML – Nasreddine Aug 05 '15 at 07:48
  • 3
    Probably, pass a different User-Agent header with your request. https://en.wikipedia.org/wiki/User_agent#User_agent_identification, see here for how http://stackoverflow.com/questions/11841540/setting-the-user-agent-header-for-a-webclient-request – Jodrell Aug 05 '15 at 08:02
  • Yeah that works, but now the website wants me to accept cookies – Photonic Aug 05 '15 at 08:06
  • Well, I don't want to get into a detailed comment discussion where I coach you harvesting data from web sites. You need to ask questions on StackOverflow. Be sure to vote up the question and answer I linked. – Jodrell Aug 05 '15 at 08:10
  • 1
    http://stackoverflow.com/questions/1777221/using-cookiecontainer-with-webclient-class, seek and ye shall find, likewise, vote them up. – Jodrell Aug 05 '15 at 08:14

2 Answers2

0

Try to use the Navigate overload with the user agent:

string useragent = "Mozilla/5.0 (Windows NT 6.0; rv:39.0) Gecko/20100101 Firefox/39.0" ;
webBrowser.Navigate(url, null, null,useragent) ;

An alternative is to use another WebBrowser such as awesomium

Graffito
  • 1,658
  • 1
  • 11
  • 10
  • With Firefox or IE 10 that work for your site, connect to this [url](http://id.furud.net/index.php) to get an accepted user agent string and try it in your code. – Graffito Aug 05 '15 at 08:21
0

After looking into passing a fake useragent (from Jodrell) in httpWebrequest, this works but i had to deal with cookies so that can get complicated. Graffito suggest to overload useragent within a webBrowser but didn't work as it gave me lots of JS loading error, this is because within that website it-self it requires a proper modern browser for it to work.

I found out that my IE itself is a version 9, so i upgraded it IE.11. Then tried Graffito solution again, but that didn't work.

So in the end i thought i might as well update webBrowser to the correct version by following this article

Community
  • 1
  • 1
Photonic
  • 1,316
  • 19
  • 31