Using PhantomJS for headless browser in Selenium - Python

Question

I'm currently using Python-Selenium to run some tests. It's supposed to log in the webpage, input username and password, and do some other things. When I execute it using the Firefox browser, it works fine, however when I use PhantonJS I'm getting the following error:

2016-01-29 16:18:29 - ERROR - An exception occurred Message: {"errorMessage":"Unable to find element with id 'user_email'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"91","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:57257","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"id\", \"sessionId\": \"fcd54080-c6e6-11e5-94bd-27954be890c7\", \"value\": \"user_email\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/fcd54080-c6e6-11e5-94bd-27954be890c7/element"}}
Screenshot: available via screen
Traceback (most recent call last):
  File "webcrawler.py", line 105, in login
    email = self.singleton.driver.find_element_by_id("user_email")
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 234, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 712, in find_element
    {'using': by, 'value': value})['value']
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 201, in execute
    self.error_handler.check_response(response)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 181, in check_response
    raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: {"errorMessage":"Unable to find element with id 'user_email'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"91","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:57257","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"id\", \"sessionId\": \"fcd54080-c6e6-11e5-94bd-27954be890c7\", \"value\": \"user_email\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/fcd54080-c6e6-11e5-94bd-27954be890c7/element"}}
Screenshot: available via screen

Debugging the issue a little more, I find the page content is:

DEBUG - <html><head></head><body></body></html>

This is I start the Selenium driver:

def __init__(self, browser="phantomjs"):
        if browser == "phantomjs":
            self.driver = webdriver.PhantomJS()
            self.driver.set_window_size(1120, 550)
        elif browser == "firefox":
            self.driver = webdriver.Firefox()
        elif browser == "chrome":
            self.driver = webdriver.Chrome()
        elif browser == "remote":
            self.driver = webdriver.Remote()
        elif browser == "ie":
            self.driver = webdriver.Ie()
        else:
            raise ValueError("Invalid browser value:  %s"  % browser)

Could someone tell me how to fix that? It works perfectly using Firefox, but I'm going to deploy this in a node in AWS so I need to use some some of headless browser.

score 2 · Accepted Answer · answered Jan 30 '16 at 00:29

2

The option given by @alecxe is valid too. In my particular case, the webpage uses SSL, so the solution is to create the PhantomJS as:

self.driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any'])

Also make sure to set the window size, in order to create a dummy browser:

self.driver.set_window_size(1120, 550)

answered Jan 30 '16 at 00:29

cybertextron

10,547
28
104
208

Yeah, I remember using that `--ignore-ssl-errors` flag to tackle different PhantomJS-specific issues too. Thanks. – alecxe Jan 30 '16 at 00:37

score 1 · Answer 2 · edited May 23 '17 at 12:19

First of all, you can still use Firefox on AWS, just run it under a virtual display:

How do I run Selenium in Xvfb?

As for your PhantomJS specific issue, you need a wait - wait for the presence of the element:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebdriverWait(driver, 10)

email_element = wait.until(EC.presence_of_element_located((By.ID, "user_email")))
email_element.send_keys(email)

Using PhantomJS for headless browser in Selenium - Python

2 Answers2