Parse html table using Nokogiri and Mechanize

Question

Using the following code I am trying to scrape a call log from our phone provider's web application to enter the info into my Ruby on Rails application.

desc "Import incoming calls"
task :fetch_incomingcalls => :environment do

    # Logs into manage.phoneprovider.co.uk and retrieved list of incoming calls.
    require 'rubygems'
    require 'mechanize'
    require 'logger'

    # Create a new mechanize object
    agent = Mechanize.new { |a| a.log = Logger.new(STDERR) }

    # Load the Phone Provider website
    page = agent.get("https://manage.phoneprovider.co.uk/login")

    # Select the first form
    form = agent.page.forms.first
    form.username = 'username
    form.password = 'password

    # Submit the form
    page = form.submit form.buttons.first

    # Click on link called Call Logs
    page = agent.page.link_with(:text => "Call Logs").click

    # Click on link called Incoming Calls
    page = agent.page.link_with(:text => "Incoming Calls").click

    # Prints out table rows
    # puts doc.css('table > tr')

    # Print out the body as a test
    # puts page.body

end

As you can see from the last five lines, I have tested that the 'puts page.body' works successfully and the above code works. It successfully logs in and then navigates to Call Logs followed by Incoming Calls.The incoming call table looks like this:

| Timestamp    |    Source    |    Destination    |    Duration    |
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |    
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |    
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |    
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |

Which is generated from the following code:

<thead>
<tr>
<td>Timestamp</td>
<td>Source</td>
<td>Destination</td>
<td>Duration</td>
<td>Cost</td>
<td class='centre'>Recording</td>
</tr>
</thead>
<tbody>
<tr class='o'>
<tr>
<td>03 Jan 13:40</td>
<td>12345678</td>
<td>12345679</td>
<td>00:01:14</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='e'>
<tr>
<td>30 Dec 20:31</td>
<td>12345678</td>
<td>12345679</td>
<td>00:02:52</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='o'>
<tr>
<td>24 Dec 00:03</td>
<td>12345678</td>
<td>12345679</td>
<td>00:00:09</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='e'>
<tr>
<td>23 Dec 14:56</td>
<td>12345678</td>
<td>12345679</td>
<td>00:00:07</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='o'>
<tr>
<td>21 Dec 13:26</td>
<td>07793770851</td>
<td>12345679</td>
<td>00:00:26</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>

I'm trying to work out how to selects just the cells I want (Timestamp, Source, Destination and Duration) and output those. I can then worry about outputting them to the database rather than in Terminal.

I have tried using Selector Gadget but it just show either 'td' or 'tr:nth-child(6) td , tr:nth-child(2) td' if I select multiple.

Any help or pointers would be appreciated!

score 10 · Accepted Answer · answered Jan 06 '12 at 20:03

There is a pattern in the table that is easy to leverage using XPath. The <tr> tag of rows with the required information lack the class attribute. Fortunately, XPath provides some simple logical operations including not(). This provides just the functionality we need.

Once we've reduced the number of rows we're dealing with, we can iterate over the rows and extract the text of the necessary columns by using XPath's element[n] selector. One important note here is that XPath counts elements starting from 1, so the first column of a table row would be td[1].

Example code using Nokogiri (and specs):

require "rspec"
require "nokogiri"

HTML = <<HTML
<table>
  <thead>
    <tr>
      <td>
        Timestamp
      </td>
      <td>
        Source
      </td>
      <td>
        Destination
      </td>
      <td>
        Duration
      </td>
      <td>
        Cost
      </td>
      <td class='centre'>
        Recording
      </td>
    </tr>
  </thead>
  <tbody>
    <tr class='o'>
      <td></td>
    </tr>
    <tr>
      <td>
        03 Jan 13:40
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:01:14
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='e'>
      <td></td>
    </tr>
    <tr>
      <td>
        30 Dec 20:31
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:02:52
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='o'>
      <td></td>
    </tr>
    <tr>
      <td>
        24 Dec 00:03
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:00:09
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='e'>
      <td></td>
    </tr>
    <tr>
      <td>
        23 Dec 14:56
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:00:07
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='o'>
      <td></td>
    </tr>
    <tr>
      <td>
        21 Dec 13:26
      </td>
      <td>
        07793770851
      </td>
      <td>
        12345679
      </td>
      <td>
        00:00:26
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
  </tbody>
</table>
HTML

class TableExtractor  
  def extract_data html
    Nokogiri::HTML(html).xpath("//table/tbody/tr[not(@class)]").collect do |row|
      timestamp   = row.at("td[1]").text.strip
      source      = row.at("td[2]").text.strip
      destination = row.at("td[3]").text.strip
      duration    = row.at("td[4]").text.strip
      {:timestamp => timestamp, :source => source, :destination => destination, :duration => duration}
    end
  end
end

describe TableExtractor do
  before(:all) do
    @html = HTML
  end

  it "should extract the timestamp properly" do
    subject.extract_data(@html)[0][:timestamp].should eq "03 Jan 13:40"
  end

  it "should extract the source properly" do
    subject.extract_data(@html)[0][:source].should eq "12345678"
  end

  it "should extract the destination properly" do
    subject.extract_data(@html)[0][:destination].should eq "12345679"
  end

  it "should extract the duration properly" do
    subject.extract_data(@html)[0][:duration].should eq "00:01:14"
  end

  it "should extract all informational rows" do
    subject.extract_data(@html).count.should eq 5
  end
end

I'm unsure how I would implement this into the code I already have, if you see the following Gist of how I imagine it should be..https://gist.github.com/1574942 — dannymcc, Jan 07 '12 at 14:53
Didn't notice your reply until just now. I've [forked your gist and added some code](https://gist.github.com/1592493). I've also answered your other question regarding this problem. — ezkl, Jan 11 '12 at 02:03

score 2 · Answer 2 · edited May 23 '17 at 11:46

2

Your answer lies in this railscasts

http://railscasts.com/episodes/190-screen-scraping-with-nokogiri

This too can help

How do I parse an HTML table with Nokogiri?

edited May 23 '17 at 11:46

Community

1
1

answered Jan 09 '12 at 13:06

dbKooper

1,035
10
17

score -1 · Answer 3 · answered Jan 06 '12 at 06:50

-1

You should be able to reach the exact node you required from the root (worst case) using XPath selectors. Using XPath with Nokogiri is listed here.

For detail on how reach all your elements using XPath, look here.

answered Jan 06 '12 at 06:50

jake

2,371
1
20
31

Is the documentation you linked with only for parsing xml data, or will it also work with html webpages? – dannymcc Jan 06 '12 at 17:59
Yes.Check this one too nokogiri.org/tutorials/searching_a_xml_html_document.html – jake Jan 06 '12 at 19:50

Parse html table using Nokogiri and Mechanize

3 Answers3