5

I am trying to scrape some info from a website using python selenium. However, when I submit the form, the result page keeps showing me "Your web browser requires JavaScript to access the page". Can anyone let me know how to solve this?

Please see below for my code:

from selenium import webdriver

profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0")
profile.set_preference("javascript.enabled", True)
broswer = webdriver.Firefox(profile)
broswer.get(http://www.cathaypacific.com/cx/en_CA.html)

input = broswer.find_element_by_id('depart-label')
input.clear()
input.send_keys('Hong')

WebDriverWait(broswer, 10, poll_frequency=0.1).until(lambda drv:    len(drv.find_elements_by_css_selector("ul.ui-autocomplete li")) > 0)
broswer.find_element_by_css_selector("ul.ui-autocomplete li").click()

time.sleep(3)
input = broswer.find_element_by_id('destination-label')
input.send_keys('van')
WebDriverWait(broswer, 10, poll_frequency=0.1).until(lambda drv: len(drv.find_elements_by_css_selector("ul.ui-autocomplete li")) > 0)
broswer.find_element_by_css_selector("#ui-id-2 li").click()

broswer.find_element_by_class_name("button-submit").click()
Stanley Li
  • 165
  • 1
  • 1
  • 9
  • what are the versions of Selenium and FF? I have no problems with executing JS inside the browser even w/o explicit enable – user3159253 Feb 23 '16 at 01:01
  • I am using python2.7; selenium 2.52.0; firefox 38.6.0 – Stanley Li Feb 23 '16 at 01:09
  • @user3159253 I can execute JS inside the browser too w/o explicit enable. However, for some reason when I try to get info from the above website, the return html asks me to enable javascript – Stanley Li Feb 23 '16 at 01:10

1 Answers1

4

There is nothing wrong with the code itself. It worked for me for the first time.

Now, the real problem is that this particular website is using a third-party anti web-scraping service called Distil Networks, which apparently has a way of detecting a selenium-webdriver powered browser, more information here:

Community
  • 1
  • 1
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195