Why selenium is not starting next url after quiting the previous url

Question

Actully I'm working on some URLs. URLs are store in my database while picking one by one and looking for it resources and storing that resources in a database. if I do this with out using driver.quit() somehow for everyurl it store information from the beginning so I decide to use driver.quit now it work only for the first URL and does not work on second URL and so on. But currently I'm just printing the data to see it works according to me. already check How to loop through a list of urls using Selenium and Python

below is the part of code::

def select_url_test():
    http = 0
    https = 0

    driver = webdriver.Chrome()

    db_conn1 = Foreign_Key_table.database_conn()
    db_conn1.execute("SELECT url,id FROM SELECTED_URLS WHERE url_status = 'VALID' AND captcha_status = 'NO' LIMIT 2")
    urls = db_conn1.fetchall()

    for url_aa in urls:
        full_url = 'https://' + url_aa[0]  # url is in form of ('google.com' ,)
        # print(url[0])
        # print(full_url)
        driver.get(full_url)
        time.sleep(2)
        for request in driver.requests:    
            if request.url.startswith('https'):
                https += 1            
                print(request.url)
        driver.quit()

is there a need to put `driver.quit()` in the for loop? you can move this statement outside of for loop. i.e launch the driver , loop all the URLs, and then quit at last. — simpleApp, Nov 05 '22 at 14:25
@simpleApp alreday tried this then actually it start for everyurl from the begining i.e list is `['google.com',facebook.com]` for `google.com` it works for when it comes to facebook.com it print again google.com and facebook.com data — Osama Billah, Nov 05 '22 at 14:30
how ```urls``` looks like ? and also could not find ```driver.requests``` pls post imports from your script. — simpleApp, Nov 05 '22 at 14:42

score 1 · Accepted Answer · answered Nov 05 '22 at 16:21

You need to put the line driver = webdriver.Chrome() into the loop. That's the line that actually opens chrome. So, at the end of the loop you do driver.quit(). That closes the browser. And at the beginning of each loop you need to open it again. So place driver = webdriver.Chrome() before driver.get(full_url) inside the loop.

Why selenium is not starting next url after quiting the previous url

1 Answers1