Use a web page as an input for a C# project

Question

How do I scan (as in scanf (from C) and the Scanner class (from Java)) in a webpage's content (html) and use it as the input for my program?

string[] line = new string[length];
for (int i = 0; i < line.Length; i++) {
  line[i] = Console.ReadLine();
  //Can I have ReadLine read from a website not the Console?
}

e.g. I want to put a text file containing the schedules for the bus near me on web server, then access this and use it to generate the output for my application. This way I can update it and always have access to these updates.

P.S. I'm a beginning programmer, and particularly a beginning C# programmer, so I'm having trouble finding what I'm looking for as I don't know quite what to search for.

Thanks for the help: I was able to start searching for the right thing and this worked:

class MainClass
{
    public static void Main (string[] args)
    {
        // Create web client.
        WebClient client = new WebClient();

        // Download string.
        string value = client.DownloadString("http://www.example.com");

        // Write values.
        Console.WriteLine("--- WebClient result ---");
        Console.WriteLine(value.Length);
        Console.WriteLine(value);
    }
}

Search terms: "web scraping" and "C# WebClient download". Please do research and than ask concrete questions with sample code. — Alexei Levenkov, Jan 23 '13 at 19:37

score 0 · Answer 1 · edited May 23 '17 at 12:12

0

You can request the content by a URL and then load it into an HTML document using something like the HTML Agility Pack. See this SO thread for suggestions What is the best way to parse html in C#?

edited May 23 '17 at 12:12

Community

1
1

answered Jan 23 '13 at 19:37

Eli Gassert

9,745
3
30
39

score 0 · Accepted Answer · answered Jan 23 '13 at 19:44

private void Test()
{
    string pageData = DownloadWebPage(new Uri("http://www.yourftp.com"));
    //parse data
}

private string DownloadWebPage(Uri path)
{
    string webPageData = null;

    using (WebClient client = new WebClient())
    {
        client.DownloadStringAsync(path);
        client.DownloadStringCompleted += (sender, args) => webPageData = args.Result;
    }

    return webPageData;
}

You can use the method above to download the text file from your web server once it has uploaded. Just pass the URI to the DownloadWebPage method and it will return the text page. If you want help parsing the text document to provide meaningful C# representations you will need to give an example of the text file and describe how you want to parse it.

Thanks! I was able to follow a search for WebClient and find what I needed. — , Jan 23 '13 at 22:00

Use a web page as an input for a C# project

2 Answers2