I am writing a program that opens 15 Chrome Selenium instances using multiprocessing module. Each session refreshes itself and extracts some text every five seconds. Here is the code :
import os
import time
import multiprocessing
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
def driver_routine(url, file_session):
chrome_options = webdriver.ChromeOptions();
chrome_options.add_argument(r'--user-data-dir=' + os.path.dirname(os.path.abspath(__file__)) + '\\' + file_session + '\\selenium')
driver = webdriver.Chrome(ChromeDriverManager().install(), options = chrome_options)
while True:
driver.get(url)
text = driver.find_element_by_xpath('//body')
time.sleep(5)
if __name__=='__main__':
processes = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
url = 'https://somesite.com/'
for i in range(len(processes)):
file_session = 'session-' + str(i)
processes[i] = multiprocessing.Process(target=driver_routine, args=(url, file_session,))
processes[i].start()
The issue is that it fills up my CPU memory (from task manager) to 100%, making my PC work very slow. I figured that adding these two lines in the driver_routine function:
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
will decrease the CPU memory usage. So that kind of fixed my issue. But now the problem is that I can't see the browser instances. Is there any way to satisfy both of my issues? I was thinking maybe starting the browser in --headless mode and reopening its GUI form (somehow) whenever I feel the need, that would be awesome. Or maybe a work around regarding the memory usage issue. I am a beginner in python and still learning. Any help is appreciated.
