3

Why does this happen using selenium with python and firefox? Selenium cant scale to a simple loop. Why cant firefox scale? Over time it decides to thow an error and quit.

Mozilla Firefox 45.0 selenium.version '2.53.2'

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
mylist = ['page1','page1',...,'pageN']
while True:
   for data in mylist:

        browser = webdriver.Firefox()
        try:
           myfunc(data) # Code that uses selenum to fetch pages
        except:
            pass
        browser.quit()
   time.sleep(60)



No handlers could be found for logger "sentry.errors.serializer"
Traceback (most recent call last):
  File "platrieveerp.py", line 231, in <module>
    browser = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/webdriver.py", line 103, in __init__
    self.binary, timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable
    % (self.profile.path))
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpOgsKOx If you specified a log_file in the FirefoxBinary constructor, check it for details.
Tampa
  • 75,446
  • 119
  • 278
  • 425
  • Quick check - would the same error be thrown in case of `driver.close()` instead of `driver.quit()`? Thanks. – alecxe May 22 '16 at 23:52

1 Answers1

1

Selenium is not failing while running the loop, it is failing at webdriver.Firefox(). The error is recognized when the connection times out at a certain time after line 231 when the script still cant recognize an open firefox application.

It is just a bug in selenium when running with a headless browser. To get a working version of selenium, run 

sudo pip install -U selenium

This should be a similar issue: Selenium: FirefoxProfile exception Can't load the profile

Community
  • 1
  • 1
dan arters
  • 208
  • 2
  • 10
  • I actually had the same issue earlier today and thought I resolved it with the above statement. It turns out that it was only working when I run the script through a crontab instead of in the terminal. Not that running your script with a crontab is the solution, but just sharing my same problem – dan arters May 23 '16 at 03:54
  • I ran the script twice today, successfully and then started falling to this trap, cleared temp files, did not work. Worst part is, it is still working fin in my local, just stopped working on the server! – Shaardool Jan 26 '17 at 10:45