7

Have a python script running some scrapers using selenium and chromedriver.

Have been scraping the same sites for a few years now with no issues. Starting last night, the same sites have started to load EXTREMELY slowly when loaded through chromedriver, though loading on my regular un-automated browser is totally fine. I've tried uninstalling and reinstalling chromedriver, upgrading, restarting, etc. to no avail. This has happened across two completely separate sites, both became slow starting last night. I am not blocked from the sites, but they load much slower than anything else.

It almost feels like a memory allocation issue, as even javascript and scrolling performs much slower than it used to. But I changed no code, and the issue arose even without an update to chromedriver (used to be fast on 112, but then became slow last night while still on version 112).

Using Selenium 4.2.0, and ChromeDriver 113.0.5672.63, though I was on version 112 yesterday and still seeing the error.

Does anyone know if there was a widespread change or something I'm not aware of?

snn
  • 405
  • 5
  • 15
  • Selenium 4.2.0 is a bit old. Have you tried 4.9.0? – CodeMonkey May 05 '23 at 23:55
  • @CodeMonkey just upgraded to 4.9.0. everything runs fine, but the issue still remains. painfully slow and unable to really process anything in a reasonable way. – snn May 06 '23 at 02:28
  • Not sure how much this might help your particular situtation but headless browsers and drivers can make your scripts faster. – CodeMonkey May 06 '23 at 16:38
  • @CodeMonkey I think due to the angular dynamic loading and other anti-scraping measures it's not really possible with headless – snn May 06 '23 at 21:06
  • I've been seeing similar behavior. I've noticed chrome doesn't feel as snappy as it used to. This started within the last week or so. That said, I have a few chrome_options (from years of bug workarounds) that I have yet to eliminate to see if any are causing the issue with the latest build. – Marcel Wilson May 17 '23 at 17:19
  • 1
    I have a test suite that takes 7 minutes in chrome but only 4 minutes in edge (older version of chromium) using identical browser options. – Marcel Wilson May 17 '23 at 20:46
  • I have the same issue with 4.9.1 – Yaryna May 18 '23 at 23:31
  • I'm having the same issue. I'm on Selenium 4.4.3 and the newest version of Chrome (114.0.5735.133) and the newest version of Chromedriver (114.0.5735.90). I see that Chromedriver is launching the wrong architecture but like you I cannot find a way to fix it. I tried upgrading to Selenium 4.10 but it didn't make a difference so I downgraded back. I need to be able to run some of these scrapers in headed mode as well. Hopefully we can figure out a solution soon. – bogenton Jun 22 '23 at 17:36

3 Answers3

1

it happening because chromedriver started version with wrong architecture, in my case chromedriver started version x86_x64 when my Mac using arm64 architecture

I don't know how to fix it, maybe someone knows
enter image description here

uninstall the current JDK version and install the latest arm version from oracle.com/java/technologies/downloads/#jdk20-mac (without usage brew)

VVV
  • 55
  • 6
  • Hi I've tried this, I'm running JDK 20, but it's still not working. Still doing the x86 version. Have you got any workarounds? – OssenCoder Jun 12 '23 at 15:00
  • definitely seems like the likely culprit here - would love to understand how to properly resolve in a maintainable way – snn Jun 14 '23 at 23:30
0

Have you tried downgrading your chrome? I use webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options) to auto-download chrome drivers on my script based on my chrome version. I checked that my chrome was updated 2 days ago and my script became slow to even load pages. My chrome updated to new stable 113.0.5672.92/93 and as I downgraded to an old stable version, all my scripts started working normally as before. Use following guide to downgrade:

https://helpdeskgeek.com/how-to/how-to-downgrade-google-chrome-to-an-older-version/

If this does not work, wait for few days to get official release of new stable chrome_driver which might solve the issue.

Krunal Dholakia
  • 11
  • 2
0

I also had this issue with my Amazon Corretto 11 which Intellij downloaded automatically on my ARM architecture. When printing System.getProperty("os.arch") I was getting x86_x64 instead of aarm64.

I've removed corretto and installed OpenJDK11 version and issue is fixed.

Adriaan
  • 17,741
  • 7
  • 42
  • 75
Marko
  • 1
  • 1