read website information, display application

Question

If it is possible to read information from a website that is organized using a standardized layout, meaning that the site's controls/textboxes/button etc are always in the same location, but the data/values it is holding changes.

In a C# WinForm application can I open that page in the background read some values and use those in my form? Is there a way to reference specific areas/items on the webpage, even if it is something as inefficient as tabbing exactly 12 times? Am I totally dreaming here?

Again I don't need to click certain things on the page, just read what is in a certain textbox or things of that nature.

Take alook at this http://stackoverflow.com/questions/26857/how-do-you-programmatically-fill-in-a-form-and-post-a-web-page — Micah Armantrout, Mar 15 '12 at 01:12
"screen-scraping" is the term you are looking for..and yes, its easy and common to do. — E.J. Brennan, Mar 15 '12 at 01:21

score 6 · Answer 1 · edited Nov 27 '17 at 15:04

6

Html Agility Pack is a popular choice for doing this kind of thing.

edited Nov 27 '17 at 15:04

carla

1,970
1
31
44

answered Mar 15 '12 at 01:16

David Peden

17,596
6
52
72

Never knew that existed thats awsome! – Micah Armantrout Mar 15 '12 at 01:20

score 4 · Accepted Answer · answered Mar 15 '12 at 01:25

You could also use the webbrowser control to do this. To get all of the posters in this thread and their reputation, you could do this:

private void Form1_Load(object sender, EventArgs e)
{
    webBrowser1.Navigated += new WebBrowserNavigatedEventHandler(webBrowser1_Navigated);
    webBrowser1.Navigate("http://stackoverflow.com/questions/9712699/read-website-information-display-application");    
}


private void webBrowser1_Navigated(object sender, WebBrowserNavigatedEventArgs e)
{

    foreach (HtmlElement ele in webBrowser1.Document.GetElementsByTagName("SPAN"))
    {
        if (ele.GetAttribute("title") == "reputation score")
        {  
            MessageBox.Show(ele.Parent.Children[0].InnerText + " - "+ ele.InnerHtml);
        }
    }
}

score 1 · Answer 3 · answered Mar 15 '12 at 01:14

1

Sure, you can do this. The exact implementation might change based on the web page, it's layout, etc.

As a basic/simple outline: Use a WebClient to retrieve the web page as a string and then use a Regex to read the matching HTML part. Things like "hit tab x times" won't work and are rather hard or complicated to implement as you'd have to either embed a browser control or parse the HTML yourself.

answered Mar 15 '12 at 01:14

Mario

35,726
5
62
78

+1. Note that RegEx is useful for picking values from the page when there is obvious chunks to locate them like "id='aaa'>text to scrap<" . If you need to find "second span inside third nested div" - time to use HtmlAgilityPack as suggested by DPeden. – Alexei Levenkov Mar 15 '12 at 01:22

read website information, display application

3 Answers3