Scrape website data without opening the browser (python)

Question

I want to iteratively search for 30+ items through a search button in webpage and scrape the related data.

My search items are stored in a list: vol_list

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome("driver path")
driver.get("web url")

for item in vol_list :
mc_search_box = driver.find_element_by_name("search_str")
mc_search_box.clear()
search_box.send_keys(item)
search_box.send_keys(Keys.RETURN)

After search is complete I will proceed to scrape the data for each item and store in array/list. Is it possible to repeat this process without opening browser for every item in the loop?

See [this SO answer](http://stackoverflow.com/questions/7593611/selenium-testing-without-browser)-- especially the one by Stéphane Bruckert — erapert, Feb 24 '17 at 18:59

score 1 · Answer 1 · answered Feb 28 '17 at 18:19

1

You can't use chrome and other browsers without opening it.

In your case, headless browsers should do the job. Headless browsers simulates browser, but doesn't have GUI.

Try ghost driver/ html unit driver/ NodeJS. Then you will have to modify at least this line with the driver you want to use:

driver = webdriver.Chrome("driver path")

Good luck!

answered Feb 28 '17 at 18:19

vilkg

625
6
7

1

Thanks! I have started using PhantomJS() – Stelios Mar 01 '17 at 09:05

score 0 · Answer 2 · answered Jul 09 '22 at 07:23

If you're using firefox, you can apply the headless option:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)
driver.get('your url')

Scrape website data without opening the browser (python)

2 Answers2