scraping the text from source code using python

Question

I'm trying to scrape google search results using python and selenium. I'm able to get only the first search result. Here is the code I'm using.

        driver.get(url)
        res = driver.find_elements_by_css_selector('div.g')
        link = res[0].find_element_by_tag_name("a")
        href = link.get_attribute("href")

How can I get all the search results?

What have you tried to get more than one result? – JeffC Jul 19 '17 at 01:55 — JeffC, Jul 19 '17 at 01:55
I utilized the solution proposed by Andersson. – Joe Jul 19 '17 at 22:45 — Joe, Jul 19 '17 at 22:45

score 0 · Accepted Answer · answered Jul 18 '17 at 18:05

Try to get list of links (from first page only. If you need to scrape more pages, you need to click "Next" button in a loop and append results from following pages) as below:

href = [link.get_attribute("href") for link in driver.find_elements_by_css_selector('div.g a')]

P.S. You also might use solutions from this question to get results as GET request response with requests lib

scraping the text from source code using python

1 Answers1