0

I'm working on a project that uses Selenium Python for web scraping. I want to use the multiprocessing library to run multiple windows at once BUT I want to set a limit for simultaneously running browsers.

My idea is to have five processes, of which two are running simultaneously at a time. Once one is finished, a new one can start. I searched for this online, but every solution I tried resulted in either one process running or all running at once.

Does anyone have experiences with multiprocessing webdrivers?

jacksc0tt
  • 11
  • 4
  • Create a multithreading pool of size 2 (note that since selenium runs in its own process, it should be sufficient to use multithreading). In particular, look at my answer to ensure that the driver instances are terminated properly. If you have extensive calculations on scraped results, then pass to the worker function a multiprocessing pool to which such tasks can be submitted. – Booboo Oct 10 '22 at 11:56

0 Answers0