Run Multithreaded Selenium Automation with multiple browser instances

Question

Im trying to upload files i have made a function which does this job, im trying to make the program to run multithread and upload for example 10 files instead of 1.

This my code i have tried clean it and only leave the important part and clarify the structure. im trying to find a way to open more browser instances and upload the files faster.

def upload():

    # DRIVERS
    options = webdriver.ChromeOptions()
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    driver = webdriver.Chrome(executable_path=path, options = options)


    # ACTION START
    driver.get(url)         
    time.sleep(2)

    # UPLOAD CODE FOR 1 FILE
    # I USE THE VARIABLE N TO LOOP THROUGH THE FILES 
    # ALL FILES ARE NAMED FROM 0 UP
 

    # FILE UPLOADED.. SLEEPING 
    time.sleep(1200)

    driver.quit()
    obj = driver.switch_to.alert
    obj.accept


# FILES LIST IN A TEXT FORMAT TO LOOP THROUGH
with open(r'path\for\fileslist') as f:
            lines = f.read().splitlines()
threads = []

for line in lines:
    t = threading.Thread(target=upload, args=(webdriver.Chrome(path),))
    t.start()
    threads.append(t)

    # ADD 1 EACH TIME TO UPLOAD THE NEXT FILE 
    N += 1


for thread in threads:
    thread.join()

score 0 · Answer 1 · answered Jul 31 '21 at 18:46

Your case is quite specific. You should be using multiprocessing module instead of threading for this purpose. Or better yet instead of having multiple chrome selenium windows, open 10 tabs with the same url and start uploading in all of them. You can find how to use this functionality here

If you need a confirmation of upload after, you might want to use multiprocessing so each process handles a seperate tab

Make sure you have enough CPU and RAM since selenium likes spiking up those two quite often

Run Multithreaded Selenium Automation with multiple browser instances

1 Answers1