Visiting an other URL at the server by code and extracting some data from there

Question

I am working in Asp.net MVC and have a peculiar requirement for which I have no idea.

What I want is that when a user request a particular URL from my site, I want to visit some preset URL in the database and extract some data and bind them to View before rendering.

For example, If you visit mysite.com/Search/Index, then in my action method Index, i want to visit the anothersite.com/someparticular/url, extract the value in <div> with id="searclbl", bind it to my view and render the page.

I need to read the HTML because the sites I am working with don't offer any Web services or RSS.

Any sort of help or guidance in this matter is appreciated.

score 2 · Accepted Answer · answered Jan 07 '12 at 12:15

I believe you might be able to pull this off using HtmlAgilityPack (which can be installed via a NuGet package inside your project).

For example: Let’s assume your Index View of the SearchController is strongly typed to the following ViewModel:

public class SearchViewModel
{
    public string DivElement { get; set; }
    //other properties...
}

This is the Index ActionResult():

public ActionResult Index()
{
    var model = new SearchViewModel();
    model.DivElement = GetDivFromWebSite();

    return View(model);
}

The GetDivFromWebSite() method is where I use HtmlAgilityPack to fetch information from another web site and is defined like so:

private string GetDivFromWebSite()
{
    var baseUrl = new Uri("http://www.anotherdomaine.com");
    HtmlAgilityPack.HtmlDocument document = new HtmlDocument();
    using (WebClient client = new WebClient())
    {
        document.Load(client.OpenRead(baseUrl));
    }

    if (document == null) return "nothing found!";
    var div = document.DocumentNode.SelectNodes("//div[@id='missing-category']").FirstOrDefault();

    return div.InnerHtml;
}

This might do the trick!

BTW it will be great if u can throw some light on this part `("//div[@id='missing-category']")` ... I don't understand it fully....just a little brief as to what's been done here and how can we traverse it for other divs and spans and other tags etc. — Pankaj Upadhyay, Jan 16 '12 at 11:02
Well…considering I haven’t spent a lot of time playing around with HtmlAgilityPack, my skills are somewhat limited. Perhaps the documentation (or examples) could better. Or here is a link: http://stackoverflow.com/questions/846994/how-to-use-html-agility-pack — Vlince, Jan 16 '12 at 14:37
In a nutshell, I’m first getting the html code for a given URL (aka document) then, SelectNodes() could be pretty much anything so I say: Find me all the div’s with an id attribute equal to ‘missing-category’ and I since I’m only concern with the first div it finds, I use linq FirstOrDefault() in case more than one div similar to this one is found. — Vlince, Jan 16 '12 at 14:38

Visiting an other URL at the server by code and extracting some data from there

1 Answers1