How can I count the amount of outbound links a page has?

Question

Learning scraping with Ruby. I'm trying to count the amount of outbound links a given page has, but I'm not sure how to tell Ruby I only want the outbound links counted.

My current code:

require "open-uri"

# Collect info
puts "What is your URL?"
url = gets.chomp
puts "Your URL is #{url}"
puts "Loading..."

# Check keyword count
page = open(url).read
link_total = page.scan("</a>")
# obl_count = ???
link_count = link_total.count
puts "Your site has a total of #{link_count} links."

How can I complete this?

score 2 · Accepted Answer · edited May 23 '17 at 12:11

2

Just as you should never parse HTML with regular expressions, you should probably be using nokogiri to do the dirty work for you.

In simple terms you can use CSS selectors to find tags. From there it's easy to count:

Nokogiri::HTML(page).css('a').length

edited May 23 '17 at 12:11

Community

1
1

answered Apr 11 '12 at 03:36

tadman

208,517
23
234
262

Thank you, I didn't know about this rule. Do you have any idea how I would separate the internal and outbound links? i.e. count links on specific Wiki page for total count, then subtract the links on the page which link to other Wikipedia pages? – Stn Apr 11 '12 at 03:45
1. In some (rare) cases, the `` tag is used for purposes other than links, so you need to detect `href` in addition to that. 2. You also need to do that to filter out the within-site links. 3. Often, there is more than one link to a single page. You need to `uniq` the relevant array. – sawa Apr 11 '12 at 03:52
You can inspect the contents of each link if you like. Nokogiri returns an array of special objects that can be used to access the `href` attribute: `Nokogiri::HTML(page).css('a').collect { |a| a.attr('href') }` A naïve test would be to see if the first character of a link is `/` in which case it is a local link. A more robust test would use the URI parsing module and `URI.join` to determine if it's one of the many Wikipedia sites that is being linked to. – tadman Apr 11 '12 at 03:52

score 0 · Answer 2 · answered Oct 18 '13 at 12:29

You can use:

http://anemone.rubyforge.org/

Anemone is a Ruby library that makes it quick and painless to write programs that spider a website. It provides a simple DSL for performing actions on every page of a site, skipping certain URLs, and calculating the shortest path to a given page on a site.

The multi-threaded design makes Anemone fast. The API makes it simple. And the expressiveness of Ruby makes it powerful.

How can I count the amount of outbound links a page has?

2 Answers2