0

I'm trying to setup some web automation using Selenium and I'm trying to go off of the answer here which is unfortunately a bit dated and using python 2, I'm using python 3. In the simplest terms, I'm going to be using discord's onMessage event to send url's to all my threads in which I'll start the automation. However, I'm having some issues getting this to work well with Selenium, when comparing this to other answers on here I don't see an immediate difference.

main.py:23: DeprecationWarning: use options instead of chrome_options
driver = webdriver.Chrome('./chromedriver', chrome_options=options)
Thread-1 True
Thread-1 Received https://google.com
Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stdout>'> at interpreter shutdown, possibly due to daemon threads

Thread 0x00007f173c273700 (most recent call first):
  File "main.py", line 39 in openWebpage
  File "main.py", line 34 in run
  File "/usr/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/usr/lib/python3.7/threading.py", line 885 in _bootstrap

Current thread 0x00007f173d9e8740 (most recent call first):
Aborted

Regardless of how many threads I declare, only a single webdriver opens and does not execute the get(message) to navigate to the URL.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
import threading
import time
import re
import discord
import asyncio
import json
from multiprocessing import Queue

print_lock = threading.Lock()

class MyThread(threading.Thread):

    options = Options()
    #options.add_argument('--headless')
    #options.add_argument('--disable-gpu')
    driver = webdriver.Chrome('./chromedriver', chrome_options=options)
    
    def __init__(self, queue, args=(), kwargs=None):
        threading.Thread.__init__(self, args=(), kwargs=None)
        self.queue = queue
        self.daemon = True
        self.receive_messages = args[0]

    def run(self):
        print(threading.currentThread().getName(), self.receive_messages)
        val = self.queue.get()
        self.openWebpage(val)
        
    def openWebpage(self, message):
        if self.receive_messages:
            driver.get(message)
            with print_lock:
                print(threading.currentThread().getName(), "Received {}".format(message))

def main():
    threads = []
    for t in range(5):
        q = Queue()
        threads.append(MyThread(q, args=(t % 2 == 0,)))
        threads[t].start()
        time.sleep(0.1)

    for t in threads:
        t.queue.put("https://google.com")

if __name__ == '__main__':
    main()

What am I doing wrong?

TehPirate
  • 154
  • 1
  • 12
  • You are attempting to share a single webdriver instance among multiple threads. That's not supported. Selenium is single-threaded. You can, however, instantiate a separate webdriver in each thread. – Tim Roberts Aug 16 '21 at 02:54
  • I thought initiating it the way I am would create a new instance for every thread, with the assumption each thread is a new MyThread? – TehPirate Aug 16 '21 at 02:57
  • 2
    No, you're creating one webdriver instance when you create the class. All the instances will have a `self.driver` member, but they're all referring to the same object. Move that initialization into `__init__` and you should be OK. – Tim Roberts Aug 16 '21 at 03:01

0 Answers0