2

Good day to all! I've been experiencing this problem for a week now but I don't think I can solve it and I also do not see any solution based on articles online. Hopefully someone can help me here...

My scenario: I need to monitor prices from 6 different tables in one page that changes almost every second. By end of day, I would close the browser (by pressing the X button) and terminate the script (by pressing Control+C) then run again in the morning and let it run through out the day. The script is written in python and is using selenium to read the prices. The browser I use is Chrome. My OS is Windows 2008 R2; Selenium version is 3.14.1

here is partial part of the code. It is just plainly reading the prices within the tables using find_elements_by_id inside an infinite loop with 1 second interval.

While True:
    close1 = float(browser.find_element_by_id('bnaBox1').find_elements_by_id('lastprc1')[0].text.encode('ascii','ignore'))
    close2 = float(browser.find_element_by_id('bnaBox2').find_elements_by_id('lastprc2')[0].text.encode('ascii','ignore'))
    close3 = float(browser.find_element_by_id('bnaBox3').find_elements_by_id('lastprc3')[0].text.encode('ascii','ignore'))
    close4 = float(browser.find_element_by_id('bnaBox4').find_elements_by_id('lastprc4')[0].text.encode('ascii','ignore'))
    close5 = float(browser.find_element_by_id('bnaBox5').find_elements_by_id('lastprc5')[0].text.encode('ascii','ignore'))
    close6 = float(browser.find_element_by_id('bnaBox6').find_elements_by_id('lastprc6')[0].text.encode('ascii','ignore'))
    time.sleep(1)
...

During the first few minutes of the run, the scripts consumes minimal amount of CPU (approx 20~30 percent) but after few more minutes, consumption slowly shoots up to 100%! There is no other processes running in the machine than the script.

Troubleshooting I've done so far (they all did not solve my issue)

  • upgraded my chrome to latest version - v71 and chromerdriver 2.44
  • rolled back Chrome to previous versions (v62, v68, v69, v70)
  • rolled back Chromedriver version to 2.42 and 2.43
  • cleared my %TEMP% files -
  • rebooted machine (multiple times)

The program only gets values within tables but I suspect that somewhere in the background, as the the script runs, unnecessary data is piling-up which causes the CPU to hit the ceiling.

Hoping that someone can help me figure out what causes this problem in the CPU and resolve the issue.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Errol Flynn Calado
  • 21
  • 1
  • 3

2 Answers2

3

It would be tough to guess the exact reason of 100% CPU Usage without any visibility to your code blocks specifically the WebDriver configuration. So the answer will be pretty much based on generic guidelines as follows:

  • Never close the browser (by pressing the X button). Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
  • Never terminate the script (by pressing Control+C). Incase there are presence of zombie WebDriver or Web Browser instances you can programatically remove them.

  • A couple of useful ChromeOptions() and their usage are as follows:

    options.addArguments("start-maximized"); // open Browser in maximized mode
options.addArguments("disable-infobars"); // disabling infobars
options.addArguments("--disable-extensions"); // disabling extensions
options.addArguments("--disable-gpu"); // applicable to windows os only
options.addArguments("--disable-dev-shm-usage"); // overcome limited resource problems
options.addArguments("--no-sandbox"); // Bypass OS security model

  • Using hardcoded sleeps in the form of time.sleep(1) is a big No.

  • Incase you are using Chrome in headless mode, there had been a lot of discussion going around about the unpredictable CPU and Memory Consumption by Chrome Headless sessions.
  • Always keep your Test Environment updated with the latest released binaries as follows:
    • Upgrade ChromeDriver to current ChromeDriver v2.44 level.
    • Keep Chrome version between Chrome v69-71 levels. (as per ChromeDriver v2.44 release notes)
    • Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
    • If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
    • Take a System Reboot.
    • Execute your @Test.
  • From Space and Memory Management perspective:
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • The article Free Up and Release... is actually incredibly misleading according to the comments on its own page and https://serverfault.com/questions/597115/why-drop-caches-in-linux Do not use: /proc/sys/vm/drop_caches – dreamwork801 Nov 09 '20 at 17:04
0

Have you tried releasing memory into the loop? Maybe by picking up the values (list out of the loop?) and then resetting those variables to None you can avoid excessive memory consumption.

...
while True:

...
    close1 = close2 = close3 = close4 = close5 = close6 = None

...

You can also try forcing the garbage collector:

import gc

while True: 
...
    gc.collect()

If you think that the reason may be a script another another solution to detect the problem might be to enable Chrome to do remote debug and debug the page.

--remote-debugging-port=9222

I hope some of this helps you.

angelparras
  • 12
  • 1
  • 3
  • From review: this looks more like a comment, could you please move it there? – toti08 Dec 07 '18 at 09:18
  • thanks for the suggestions. I tried releasing the memory by resetting the variables to None but it did not solve the issue. I will try forcing garbage collector and post update here... – Errol Flynn Calado Dec 07 '18 at 10:09