1

So I have some simple code that is using multithreading. It's working just fine however I'm noticing that the threads aren't being destroyed after returning the values since everytime the script runs, the Thread number in console goes up and the RAM being used also goes up after the script is done processing (which implies that something was left running after the script was done processing).

I've researched this, this, this and asking this. Can anyone suggest what could be going wrong?

if __name__ == "__main__":
def run_selenium1(a, b, c, d, e):
    
    @st.cache_data(show_spinner=False)
    def get_links(i, resumeContent):
        #stufff happens
            for something1, something2, something3, something4, something5, something6, something7 in zip(Final_Something1, Final_Something2, Final_Something3, Final_Something4, Final_Something5, Final_Something6, Final_Something7):
                Final_Array.append((something1, something2, something3, something4, something5, something6, something7))
            driver.close()
            driver.quit()
        except:
            driver.close()
            driver.quit()


    with webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options) as driver:
        try:
           #links are obtained
        except:
            driver.close()
            driver.quit()

    threads = []
    for i in links:
        t = threading.Thread(target=get_links, args=(i, Content))
        t.daemon = True
        threads.append(t)
        t.start()
    for t in threads:
        t.join()
        print("Threads destroyed") #<---- this isn't printed

EDIT: Thread Executor approach

        with ThreadPoolExecutor(max_workers=25) as executor:
        for i in links:
            task = executor.submit(get_links, i, resumeContent)
            task.join()
    executor.shutdown()
alex
  • 118
  • 1
  • 6
  • Avoid using threads yourself as much as you can try to use a thread pool if possible, such as `ThreadPoolExecutor` from the `concurrent.futures` Python module. It may works well in your use case. – Louis Lac Feb 26 '23 at 13:41
  • @LouisLac I tried this approach but it's still the same behavior. Everytime I run the script the `ThreadPoolExecutor-1` count goes up :/ I updated the questions with the approach that I took btw – alex Feb 26 '23 at 19:31
  • Does it _only_ happen with Selenium? Can you try running something different, and simpler, as the thread? Also, in your previous question on this topic, didn't you say that it was not the threads created by the pool, but pre-existing threads, that were the problem? Can you list the threads before running the pool, and after all the threads are joined, so we can see? – ProfDFrancis Feb 27 '23 at 23:17

0 Answers0