Retrieving and caching HTML from website using ASP.NET MVC 3

Question

I want a partial view that display some stuff from a website that is not under my control. The data on the website is only available through HTML, and thus I can only retrieve it by querying the web site and parsing the HTML. (The website holds a list of 50 elements, and I only want the top 10.) Now, the data from the website is not changing very frequently, so I imagine that I can retrieve the HTML on an hourly basis, and displaying a cached version on my web site.

How can I accomplish this in ASP.NET MVC 3?

score 5 · Answer 1 · answered May 03 '11 at 07:50

Ignoring the MVC3 requirement for now, you should look to using WebClient to grab the html from the website. You can do something like:

var client = new WebClient();
var html = Encoding.UTF8.GetString(client.DownloadData("http://www.somedomain.com"));

If you need to tailor your request, I'd recommend looking at HttpWebRequest, HttpWebResponse. Now that you can grab the html, you need to consider your caching mechanism, possibly in the ASP.NET runtime?

public ActionResult GetHtml()
{
    if (HttpRuntime.Cache["html"] == null)
        GetHtmlInternal();

    return Content((string)HttpRuntime.Cache["html"], "text/html");
}

private void GetHtmlInternal()
{
   var html = // get html here.

   HttpRuntime.Cache.Insert("html", html, null, DateTime.Now.AddMinutes(60), Cache.NoSlidingExpiration);
}

Thank you for a good answer. Unfortunately my reputation prohibits me from voting your post up. — kasperhj, May 03 '11 at 08:57

Emil Badh · Accepted Answer · 2011-05-03T08:48:22.600

The first solution that comes to mind is to create an action in a controller that makes an Http request to the remote web page and parses the html you want to return to your own page and then set output caching on your action.

Edit:

What controller to put the action in would depend on the structure of your web site and whether the partial view would be visible on all views or just a specific view. If the partial is visible in all views I'd either place it in the Home controller or create a "General" controller (if I anticipated more actions would go in such a controller).

If you want to manipulate the result I would probably make a model and partial view for the list. If you want to take a part of the returned html and output it as it is I would use the same method as in the answer by Matthew Abbott:

return Content(yourHtmlString);

The end would look something like this:

[OutputCache(Duration = 3600)]
public ActionResult RemoteList()
{
    var client = new WebClient();
    var html = Encoding.UTF8.GetString(client.DownloadData("http://www.somedomain.com"));

    // Do your manipulation here...

    return Content(html);
}

(Some of the above code was borrowed from the post by Matthew Abbott.)

If I want the data to display in a partial View, what would be a good Controller to put the action in? And would I need to create a Model for the View? — kasperhj, May 03 '11 at 08:19
That's great. Thank you. I think I can work out the Model-details my self. Thanks for a good answer. — kasperhj, May 03 '11 at 08:56

score 0 · Answer 3 · answered May 03 '11 at 07:45

0

You could just add OutputCache attribute on your action and set OutputCache.Duration Property to 3600 seconds (1 hour)

answered May 03 '11 at 07:45

archil

39,013
7
65
82

Retrieving and caching HTML from website using ASP.NET MVC 3

3 Answers3

Linked