Webscraping with Python

Question

I'm using this website (http://gasbuddy.com/) to collect gasoline prices. Basically, I want to write a python script that will input zip codes into the search box at the top of the page, and then scrape the results off the next page. I'm stuck at the first step, which is inputting the zip code I want into the form. This is what I have so far:

from mechanize import Browser
import urllib2

br = Browser()
baseURL = "http://www.gasbuddy.com/"
br.open(baseURL)

zipcode = "20010"

forms = [f for f in br.forms()]
print forms[0]
control = forms[0].find_control("ctl00$Content$GBZS$txtZip")
forms[0]["ctl00$Content$GBZS$txtZip"] = "20010"
br.form = forms[0]
page = br.submit()
content = page.read()
br.geturl()

Unfortunately when I submit the form, br.geturl() tells me that I haven't gotten to the page that I want (the url should look something like "http://www.washingtondcgasprices.com/index.aspx?area=Washington%20-%20NE&area=Washington%20-%20NW&area=Washington%20-%20SE&area=Washington%20-%20SW")

If you have any guidance I'd appreciate it. Thanks!

Could you give us an example on how to get to that page by navigating the site? — Fábio Diniz, Apr 03 '13 at 18:26
If you go to the search box under "Search for Local Gas Prices" and enter a zip code such as "20010" then hit the search button, it will bring you to a page with gas prices for whatever zipcode you specified. — aesir, Apr 03 '13 at 18:31
Maybe try with selenium so you actually see where it goes wrong? I had this issue many times and sometimes the website detects that you are not a real person, and displays a captcha. or things like that — nnaelle, Apr 16 '13 at 18:06
There is javascript on the page and mechanize does not support javascript. — djas, Aug 02 '13 at 15:51

score 1 · Answer 1 · edited May 23 '17 at 12:28

You can do it with Selenium:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

baseURL = "http://www.gasbuddy.com/"

browser = webdriver.Firefox()
zipcode = "20010"

browser.get(baseURL)
elem = browser.find_element_by_id("ctl00_Content_GBZS_txtZip").send_keys(zipcode)
elem = browser.find_element_by_id("ctl00_Content_GBZS_btnSearch").click()

If you want to stick to mechanize, you might want to tweak your browser a bit. But I still suspect it's the javascript that is killing you there. The solution then would be "read the javascript yourself and simulate with mechanize what it would be doing".

Webscraping with Python

1 Answers1