Driver.get a group of links?

Question

How do I use driver.get to open several URLs in Chrome.

My code:

import requests
import json
import pandas as pd
from selenium import webdriver
chromeOptions = webdriver.ChromeOptions()
chromedriver = r"C:\Users\Harrison Pollock\Downloads\Python\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(executable_path=r"C:\Users\Harrison Pollock\Downloads\Python\chromedriver_win32\chromedriver.exe",chrome_options=chromeOptions)
links = []
request1 = requests.get('https://api.beta.tab.com.au/v1/recommendation-service/featured-events?jurisdiction=NSW')
json1 = request1.json()
for n in json1['nextToGoRaces']:
    if n['meeting']['location'] in ['VIC','NSW','QLD','SA','WA','TAS','IRL']:
        links.append(n['_links']['self'])
driver.get('links')

Think of your webdriver object as a normal browser. One browser can't be on more than one page at once - you need to either use multiple tabs or multiple browsers. So what are you trying to achieve? Open all the pages at once in parallel? Or iterate through the pages? — RichEdwards, Jul 28 '20 at 05:44
Well the pages don't have to be opened at exactly the same time — onthepunt3, Jul 28 '20 at 05:47
you may need JavaScript to open `tab` if only browser can open new page in tab isntead of new window - `driver.execute_script("window.open('https://httpbin.org/get','_blank');")`. And switch tabs/windows like this `driver.switch_to.window(driver.window_handles[-1])` — furas, Jul 28 '20 at 06:23
Are you planning on driving all the tabs at the same time? Because 1 browser is only 1 process, you can't parallel run in tabs - you'll do tab a, action a, switch to tab b do action b, etc... But you can parallel multiple browsers. I wrote a browser tab manager in c# recently on here - I'll fish out a link. I see you're doing python but it's for ideas. — RichEdwards, Jul 28 '20 at 07:41
Tab management in c# https://stackoverflow.com/questions/62774094/creating-new-tabs-and-managing-them-selenium/62812045#62812045 - also uses similar code to @furas :-) If you want to give more of an outline around what you're trying to do (i.e. the problem you want to solve) we might be able to help more instead of answering your direct question. It might be we can suggest something you haven't considered — RichEdwards, Jul 28 '20 at 07:45
I am building a betting bot, which bets on various events. Some of these events can run simultaneously so I would need to run various tabs at once. Instead of using tabs can I use various windows at once? — onthepunt3, Jul 28 '20 at 08:00

score 0 · Answer 1 · answered Jul 28 '20 at 10:13

Based on the comments - you'll want a class to manage your browsers, a class for your tests, then a runner to run in parallel.

Try this:

import unittest
import time
import testtools
from selenium import webdriver

class BrowserManager:
    browsers=[]
    def createBrowser(self, url):
        browser = webdriver.Chrome()
        browser.get(url)
        self.browsers.append(browser)

    def getBrowserByPartialURL(self, url):
        for browser in self.browsers:
            if url in browser.current_url:
                return browser

    def CloseItAllDown(self):
        for browser in self.browsers:
            browser.close()



class UnitTest1(unittest.TestCase):
    def test_DoStuffOnGoogle(self):
        browser = b.getBrowserByPartialURL("google")
        #Point of this is to watch the output! you'll see this +other test intermingled (proves parallel run)
        for i in range(10):
            print(browser.current_url)
            time.sleep(1)
 
    def test_DoStuffOnYahoo(self):
        browser = b.getBrowserByPartialURL("yahoo")
        #Point of this is to watch the output! you'll see this +other test intermingled (proves parallel run)
        for i in range(10):
            print(browser.current_url)
            time.sleep(1)



#create a global variable for the brwosers
b = BrowserManager()

# To Run the tests
if __name__ == "__main__":
    ##move to an init to Create your browers
    b.createBrowser("https://www.google.com")
    b.createBrowser("https://www.yahoo.com")

    time.sleep(5) # This is so you can see both open at the same time

    suite = unittest.TestLoader().loadTestsFromTestCase(UnitTest1)
    concurrent_suite = testtools.ConcurrentStreamTestSuite(lambda: ((case, None) for case in suite))
    concurrent_suite.run(testtools.StreamResult())

This code doesn't do anything exciting - it's an example of how to manage multiple browsers and run tests in parallel. It goes to the specified urls (which you should move to an init/setup), then prints out the URL it's on 10 times.

This is how you add a browser to the manager: b.createBrowser("https://www.google.com")

This is how you retrieve your browser: browser = b.getBrowserByPartialURL("google") - note it's a partial URL so you can use the domain as a keyword.

This is the output (just the first few lines- not all of it...) - It's a print URL for google then yahoo, then google then yahoo - showing that they're running at the same time:

PS C:\Git\PythonSelenium\BrowserManager>  cd 'c:\Git\PythonSelenium'; & 'C:\Python38\python.exe' 'c:\Users\User\.vscode\extensions\ms-python.python-2020.7.96456\pythonFiles\lib\python\debugpy\launcher' '62426' '--' 'c:\Git\PythonSelenium\BrowserManager\BrowserManager.py' 
DevTools listening on ws://127.0.0.1:62436/devtools/browser/7260dee3-368c-4f21-bd59-2932f3122b2e
DevTools listening on ws://127.0.0.1:62463/devtools/browser/9a7ce919-23bd-4fee-b302-8d7481c4afcd

https://www.google.com/
https://consent.yahoo.com/collectConsent?sessionId=3_cc-session_d548b656-8315-4eef-bb1d-82fd4c6469f8&lang=en-GB&inline=false
https://www.google.com/
https://consent.yahoo.com/collectConsent?sessionId=3_cc-session_d548b656-8315-4eef-bb1d-82fd4c6469f8&lang=en-GB&inline=false
https://www.google.com/

I appreciate your response. I can't believe that you have to use two different browsers to run URLs simultaneously. I would've thought that selenium was more advanced than this. Are you sure this is the only way to run URLs simultaneously. Can they be run using different windows of the same browser? Or is there some sort of alternative to selenium? I can't be the only one with this issue? — onthepunt3, Jul 28 '20 at 15:51
That's the way it is. Multiple windows in a browser is tabs - but that's more complex and you cannot concurrently drive them (and that's a browser restriction not a selenium restriction - think that you need a rendered page to automate against... think along the lines one browser per rendering). So many reasons that it is this way that i can't cover it in a comment block. If you want to investigate alternatives you'll need to learn JS - Microsoft recently released playwright and Cypress has been knocking around for a while. They all drive browsers - but you'll hit the same hurdle. — RichEdwards, Jul 29 '20 at 09:47

Driver.get a group of links?

1 Answers1