1

I am writing a script that will save the complete contents of a web page. If I try using urllib2 and bs4 it only writes the contents of the logon page and none of the content after navigating to a search within the page. However, if I do a ctrl + s on the search results page, an html file is saved to disk that when opened in a text editor has all of the contents from the search results.

I've read several posts here on the subject and am trying to use the steps in this one:

How to save "complete webpage" not just basic html using Python

However, after installing geckodriver and setting the sys path variable I continue to get errors. Here is my limited code:

from selenium import webdriver
>>> from selenium.webdriver.common.action_chains import ActionChains
>>> from selenium.webdriver.common.keys import Keys
>>> br = webdriver.Firefox()

Here is the error:

Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python27\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 142, in __init__
    self.service.start()
  File "C:\Python27\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start
    os.path.basename(self.path), self.start_error_message)
WebDriverException: Message: 'geckodriver' executable needs to be in PATH.

And here is where I set the sys path variable:

enter image description here

I've restarted after setting sys path variable.

UPDATE:

I am now trying to use the chromdriver as this seemed more straight forward. I downloaded hromedriver_win32.zip II'm on a windows laptop) from chromedriver's download page, set the environmetal variable path to: C:\Python27\Lib\site-packages\selenium\webdriver\chrome\chromedriver.exe

but am getting the similar following error:

>>> br = webdriver.Chrome()
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python27\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 62, in __init__
    self.service.start()
  File "C:\Python27\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start
    os.path.basename(self.path), self.start_error_message)
WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
geoJshaun
  • 637
  • 2
  • 11
  • 32

1 Answers1

0

You have also to add the path of Firefox to the system variables manually, you maybe have installed firefox some other location while Selenium is trying to find firefox and launch from default location but it couldn't find. You need to provide explicitly firefox installed binary location:

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

binary = FirefoxBinary('path/to/installed firefox binary')
browser = webdriver.Firefox(firefox_binary=binary)
browser = webdriver.Firefox()
developer_hatch
  • 15,898
  • 3
  • 42
  • 75
  • @DL Still getting the same message. I'm going to settings->Advanced->Environmental Variables->Path added C:\Python27\selenium\webdriver\firefox\geckodriver-v0.17.0-win64\geckodriver.exe and C:\Program Files (x86)\Mozilla Firefox\firefox.exe – geoJshaun Jun 16 '17 at 23:37
  • @ShaunO and that's ok, but the path to firefox binary where is it? you need to search where firefox is installed and make a variable for it too :). For example take a look of it https://stackoverflow.com/questions/40208051/selenium-using-python-geckodriver-executable-needs-to-be-in-path – developer_hatch Jun 16 '17 at 23:40
  • @ShaunO also look at this guy, has a similar issue https://github.com/mozilla/geckodriver/issues/90 – developer_hatch Jun 16 '17 at 23:40
  • @DL I set the firefox_binary.py to the system and user variables called "Path" and still getting the message that the file needs to be in Path. I'm now trying chromedriver but still getting the same error. – geoJshaun Jun 19 '17 at 20:39
  • @DL I just copied over the chromedriver.exe to the Pytho27 folder and it worked. For some reason setting it as variable does not work for me. – geoJshaun Jun 19 '17 at 20:55
  • @ShaunO oh.. Thanks for letting me know! SORRY for answering late... I'm glad you solved the problem, sorry for not helping you the whole way.. – developer_hatch Jun 19 '17 at 21:31
  • I thought you already had the file in the python path! That part was the most important... Hahaha, well, problem solved! Blessings – developer_hatch Jun 19 '17 at 21:32