How to pull data from any website and use it in Windows Store apps

Question

I want to know the method of pulling the data from website and parsing it into our own code to present it to the user. For example: Consider an app in which a user types a movie name and all the poster gets fetched from various websites, like IMDb, etc. Or a user enters a movie name and all the data from IMDb is fetched. I know about certain third party API services for fetching data from IMDb, like omdbapi and imdbapi, but I want to know the method of doing so from any sort of website, not just IMDb. I am a complete newbie in this context so please guide me through this from the very beginning. I want to do this in a Windows 8 Store app using C# and XAML in Visual Studio.

Google "web scraping" and then come back to SO once you have specific questions about implementing your scraper. Also be aware that scraping somes sites for data is not allowed by their terms of service. — Troy Carlson, Apr 04 '14 at 16:55
@TroyCarlson - I know about web scraping. I just want to know how to do it in C#. I am not THAT new to this concept. Of course I have Googled quite a lot about it already. — Preetesh, Apr 04 '14 at 17:12
I have read [this thread](http://stackoverflow.com/questions/151272/given-an-imdb-movie-id-how-do-i-programmatically-get-its-poster-image) thoroughly. Please refer to this guide me. — Preetesh, Apr 04 '14 at 17:18

score 0 · Answer 1 · answered Apr 04 '14 at 17:03

0

Simple way is you use the website's RSS feed. You can find the rss feed for any website. All you have to do is pass the parameters as a query string using a web request object. the response stream will them have all the details that you want that can be parsed through in c# and worked upon.

answered Apr 04 '14 at 17:03

Karthik Ganesan

4,142
2
26
42

Not all websites provide RSS feeds. – Preetesh Apr 04 '14 at 17:05

score 0 · Answer 2 · answered Apr 04 '14 at 18:41

0

There is no standard way to do it for any website you must write your algorithm for each of the websites you want to get the content from

HttpClient is you tool in getting web content in your app

answered Apr 04 '14 at 18:41

csharpwinphonexaml

3,659
10
32
63

score 0 · Answer 3 · answered Apr 06 '14 at 11:45

0

Check out YQL: The Yahoo Query Language is an expressive SQL-like language that lets you query, filter, and join data across Web.

answered Apr 06 '14 at 11:45

catlan

25,100
8
67
78

score 0 · Answer 4 · answered Apr 06 '14 at 13:42

0

You should use Html Agility Pack.

For better performance, host your scraping service on Azure.

answered Apr 06 '14 at 13:42

Mikael Dúi Bolinder

2,080
2
19
44

How to pull data from any website and use it in Windows Store apps

4 Answers4