iteratively search for a book name in google and click the link from amazon.com

Question

I am using the following script

from selenium import webdriver
import time
import urllib.parse

browser = webdriver.Chrome()

with open("google-search-terms.adoc") as fin:
    for line_no, line in enumerate(fin):
        line = line.strip()
        query = urllib.parse.urlencode({'q': line})
        browser.execute_script(
            f"window.open('https://www.google.com/search?{query}');")

for x in range(len(browser.window_handles)):
    browser.switch_to.window(browser.window_handles[x])
    time.sleep(3)
    try:
        browser.find_elements_by_xpath(
            "//*[@id='rso']/div/div/div/a/div/cite[contains(text(),'amazon')]").click()
    except:
        pass

The input file google-search-terms.adoc contains:

The Effective Executive by Peter Drucker
The Functions of the Executive

It open multiple tabs containing search result of the texts from input file. It loop over the tabs every 3 seconds. However, it is not clicking the expected search results?

What is wrong here?

score 1 · Accepted Answer · answered Oct 07 '20 at 08:43

1

Google has a feature where you can get the result from a particular website. So the procedure here is just to search via that feature and click the first link found:

from selenium import webdriver
import time
import urllib.parse

browser = webdriver.Chrome()

with open("google-search-terms.adoc") as fin:
    for line_no, line in enumerate(fin):
        line = line.strip()
        query = urllib.parse.urlencode({'q': line + " site:amazon.com"})
        browser.execute_script(
            f"window.open('https://www.google.com/search?{query}');")

for x in range(len(browser.window_handles)):
    browser.switch_to.window(browser.window_handles[x])
    time.sleep(2)
    try:
        result = browser.find_elements_by_xpath('//div[@id="rso"]/div/div')[0] 
        result.find_element_by_xpath("./div/a").click()
    except:
        continue

answered Oct 07 '20 at 08:43

Yash

1,271
1
7
9

Thanks, hugs and kisses. – Ahmad Ismail Oct 07 '20 at 08:57
I noticed that I actually do not need to use `time.sleep(2)`. Is there any way I can concurrently click the links? If you suggest, I can create a new question for that. – Ahmad Ismail Oct 07 '20 at 09:02
1

Yes. you dont need ```time.sleep(2)``` as for simultaneous working on different tabs, i dont think it is possible. but you can enable a function of chrome via selenium where you dont wait for a page to load hence speeding up the process. [this](https://stackoverflow.com/a/56018235/7518304) will help – Yash Oct 07 '20 at 09:12

iteratively search for a book name in google and click the link from amazon.com

1 Answers1