1

I have a Python script that uses Selenium to do some web-page clicking and scraping. Script is running on Ubuntu, running on an EC2 instance. The basic code:

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import datetime
from datetime import datetime as dt
import re
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import ElementNotVisibleException
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException
    
from selenium.webdriver.common.by import By
    
#Set driver options
options = Options()
options.add_argument('--no-sandbox')
options.add_argument('--window-size=1420,1080')
options.add_argument('--headless')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-gpu')
options.add_argument("--disable-notifications")

options.add_experimental_option("excludeSwitches", ["enable-automation"])

options.add_experimental_option('useAutomationExtension', False)
    
driver = webdriver.Chrome(chrome_options=options, executable_path=r'/Applications/chromedriver_91')

# Do a bunch of stuff................

driver.quit() #invoke after web-scraping

Does driver.quit(), in this instance, essentially do the same thing as the Linux command pkill chrome? Sometimes, this script will crash because there's not enough memory. Using pkill chrome in combination with pkill -f "(chrome)?(--headless)" in the terminal itself usually kills all the processes and frees up memory, and the script will work after that.

Is driver.quit() sufficient to close all Chrome processes in headless or otherwise? Is adding something in my Python script like:

import os 
os.system("pkill chrome")

Doing anything that driver.quit() is not already doing? I just want to minimize the chances of a crash by making sure Chrome is completely closed after Python script is run.

halfer
  • 19,824
  • 17
  • 99
  • 186
DiamondJoe12
  • 1,879
  • 7
  • 33
  • 81

1 Answers1

0

driver.quit() quits (closes) only this specific driver object.
This will definitely not close any other running driver processes.
Also process involved by chromedriver appears as chromedriver or chromedriver (32 bit), not chrome. chrome process is your Chrome browser, not the Selenium webdriver.
To ensure closing the chromedriver you can use try-except-finally involving driver.quit() inside the finally block.
I'm not sure this is the optimal approach since AFAIK this may affect reporting mechanism.

Prophet
  • 32,350
  • 22
  • 54
  • 79
  • Thank you. What would you suggest I do to ensure that all chrome and chromedriver processes are killed at the end of the python script, after chromedriver does it's thing? – DiamondJoe12 Jul 22 '21 at 22:24
  • maybe to involve some kind `os.system("pkill chromedriver")` (not sure about the syntax) externally, inside the script involving the Selenium script after calling the Selenium script? I'm not sure. – Prophet Jul 22 '21 at 22:29
  • Anyway, you are asking a really interesting question. I'd like to see what will answer people really professional with OS, especially Linux OS. – Prophet Jul 22 '21 at 22:31