How to show results and still performing a task in Ruby on Rails

Question

I'm trying to implement a tool using Ruby on Rails, which should crawl a webside and search for hyperlinks. There is a problem: if the website has a huge number of links, the user needs to wait a lot of time.

This is probably a naive question: how can I show results (for example 10 results) and the crawling process still running?

Then, the user click "Next" and it shows the next 10 links, and so on.

crawl in the background, save results to db and refresh page displaying them to user periodically? — Mike Szyndel, Sep 01 '13 at 17:57
Yes, my question is: how to crawl in the background? Can I have something like "Scroll down" to load new results from DB? Thank you! — Hugo Sousa, Sep 01 '13 at 18:00
YOu asked broad, generic question so you got a broad, generic answer. Be specific! — Mike Szyndel, Sep 01 '13 at 18:21

score 0 · Answer 1 · edited May 23 '17 at 10:31

Imagine that a page has a list of links.

Implement an action in your controller that, given the a position in the links list, gives the next 10 links and returns a json from the data to be displayed.
With javascript, call the just implemented action with zero, get the json, parse it and display in the screen.
Repeat number 2 adding the number of links as parameter to the ajax call until it receives zero links

This will be way more efficient if you get all links on a page in a call, show it to the user, and then repeat to the user. Like the following:

For a given page, add an action that returns all links it have in json
Do a ajax call to that action, take the links, display, and then use each of the given links as parameter, to crawl to these links.
Do this while you have no more links. Keep a links blacklist to avoid cycles.

If you didn't got the ajax part, check the definition of ajax on wikipedia and this question

How to show results and still performing a task in Ruby on Rails

1 Answers1