3

I am wondering how to implement a solution that will retrieve data that I have scraped, and use it to display in an ASP.NET MVC web application.

The current implementation scrapes the data and displays it from the controller to the view, however by doing so, the request to view the web page will take very long due to the scraper running when a request to view the page with scraped data is processed.

Is there any implementation I can do to separate the data retrieval and the website?

Currently I have a console application scraper class that scrapes data, and a ASP.NET MVC web application that will display the data. How can I couple them together easily?

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
sutoL
  • 1,737
  • 10
  • 27
  • 48
  • Use a database to store the results of the scraping from the console app. Use the MVC Web App to query the results from the database. – Lars Kristensen Mar 16 '16 at 07:16

1 Answers1

0

Based on system size I think you can do 2 things:

  • Periodically scrape data and save it in the memory
  • Periodically scrape data and save it in the database

It is oblivious that if scrapped data is big you need to store it in database, otherwise you can keep it memory and highly boost performance.

Running tasks in asp.net periodically is covered by background workers. Some easy way to periodically run tasks is to initiate thread in Application_Start. I don't go more deeply in implementation, because it is already answered. You can reed it here: Best way to run scheduled tasks

For saving data in memory you can use something like this:

public static class Global
{
    public static ConcurrentBag<ScrapedItem> ScrapedItems;
}

*Note, it is necessary to use thread-safe collection, because of getting and adding to this collection will be done from different threads: one from background worker, one from request. Or you can use lock object when getting/setting to not thread safe collection.

Community
  • 1
  • 1
bot_insane
  • 2,545
  • 18
  • 40
  • Yes the scraped data is going to be quite big. – sutoL Mar 16 '16 at 08:27
  • In this situation your background worker should periodically update your database and your mvc application should get data from that database. In any case it is necessary to transfer data scraping logic to background worker. – bot_insane Mar 16 '16 at 08:29