pandas.read_html returns only one table

Question

I try to read ec2 pricing tables with pandas. Based on documentation I expect list of DataFrames, but got one table as a list.

Code example

import pandas
link = 'http://aws.amazon.com/ec2/pricing/' 
data = pandas.read_html(link)
print type(data)
print data[0]

Output

<type 'list'>
                               0                 1                2
0  Reserved Instance Volume Discounts               NaN              NaN
1            Total Reserved Instances  Upfront Discount  Hourly Discount
2                  Less than $250,000                0%               0%
3              $250,000 to $2,000,000                5%               5%
4            $2,000,000 to $5,000,000               10%              10%
5                More than $5,000,000        Contact Us       Contact Us

Environment:

• Ubuntu 14.10
• python 2.7.8
• pandas 0.14.1

Answer 1

http://aws.amazon.com/ec2/pricing/ uses JavaScript to fill in the data in the tables.

Unlike what you see when you point your GUI browser at the link, the data is missing if you download the HTML using urllib2:

import urllib2
response = urllib2.urlopen(link)
content = resonse.read()

(Then search the contents for <table> tags.)

To process the JavaScript you'll need an automated browser engine like Selenium, or WebKit or Spidermonkey.

Here is a solution using Selenium:

import selenium.webdriver as webdriver
import contextlib
import pandas as pd
@contextlib.contextmanager
def quitting(thing):
    yield thing
    thing.quit()

with quitting(webdriver.Firefox()) as driver:
    link = 'http://aws.amazon.com/ec2/pricing/' 
    driver.get(link)
    content = driver.page_source
    with open('/tmp/out.html', 'wb') as f:
        f.write(content.encode('utf-8'))
    data = pd.read_html(content)
    print len(data)

yields

238