Read javascript code in selenium without opening browser

Question

I'm working with selenium API to webcrapping on pages with javascript.

Is there some method to get the code without a web browser screen opens ?

I am new to this API

Is possible?

Answer 1

You have, at least, 3 basic options:

• use a headless browser, like PhantomJS, example:

  >>> from selenium import webdriver
  >>> driver = webdriver.PhantomJS()
  >>> driver.get('http://stackoverflow.com')
  >>> driver.title
  u'Stack Overflow'
  

• use a virtual display (see xvfb) with the help of pyvirtualdisplay, examples here:

  • How do I run Selenium in Xvfb?
  • Selenium with pyvirtualdisplay unable to locate element

• use a remote selenium server, either your own with setting up own nodes in a grid, or at, for example, BrowserStack, or Sauce Labs:

  >>> from selenium import webdriver
  >>> from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
  >>> 
  >>> desired_cap = {'os': 'Windows', 'os_version': 'xp', 'browser': 'IE', 'browser_version': '7.0' }
  >>> driver = webdriver.Remote(command_executor='http://username:key@hub.browserstack.com:80/wd/hub', desired_capabilities=desired_cap)
  >>> 
  >>> driver.get('http://stackoverflow.com')
  >>> driver.title
  u'Stack Overflow'
  

Answer 2

Unfortunately, you can't use JavaScript without an interpreter, which is in a browser. Though, you may use PhantomJS - a headless browser.

Answer 3

You could use headless browser. For example http://phantomjs.org/ or http://slimerjs.org/

Answer 4

"Is there some method to get the code without a web browser screen opens?"- YES! You have to import Options.

from selenium import webdriver   # for webdriver
from selenium.webdriver.support.ui import WebDriverWait  # for implicit and explict waits
from selenium.webdriver.chrome.options import Options  # for suppressing the browser

Then in the code:

option = webdriver.ChromeOptions()
option.add_argument('headless')
driver = webdriver.Chrome(options=option)

And continue with the rest of the program.