Memory increasing when webdriver running for long time using Selenium java

Question

I am using selenium java to do web page scraping, basically the app creates a WebDriver and use it all the times for all pages required(every 1 or 2 seconds it will do a get() call for a new page and extract the related content).

I am using Firefox headless mode like this:

String driverPath = this.config.getString("browser.firefox.driverPath");

FirefoxBinary firefoxBinary = new FirefoxBinary();
if (useHeadlessMode) {
    firefoxBinary.addCommandLineOptions("--headless");
}
System.setProperty("webdriver.gecko.driver", driverPath);
FirefoxOptions firefoxOptions = new FirefoxOptions();
firefoxOptions.setBinary(firefoxBinary);
webDriver = new FirefoxDriver(firefoxOptions);

I have realized that when the app running for 2 hours, it will use up to 8GB memory, and the get() call becomes extremely slow (could take around ~10 seconds).

My question is that do I miss any configuration when creating the WebDriver? Or any other solution to keep the memory usage in a low level, since I am considering to launch multiple (~100 WebDrivers) after deploying the app into the cloud.

The solution I am considering is that for a certain amount of operations, do driver.quit() for the current driver and initialize a new one. Does this sounds reasonable?

Answer 1

First of all you need to understand that whenever you launch a browser using webdriver it create a temp profile in your "Temp" directory which consumes your memory.

To avoid that you can do 2 things :

• Delete the data from "Temp" directory.
• Create a Profile for your browser and use it so that everytime you launch a browser it will not create new profile and will not consumes more memory.

Delete the data from "Temp" directory :

1. Open "Run" winodow by pressing --> "Windows key" + "r"
2. Type "%TEMP%" and press "ENTER" button
3. Select all files and Delete Permenently by "Shift + Delete"

Create a Profile for your browser :

1. First of all close the Firefox if open.
2. Open "Run" winodow by pressing --> "Windows key" + "r"

3. Type " firefox.exe –p" and press "ENTER" button Note: If it doesn't open you can try using full path enclosed in quotes.

   • On 32 bit- Windows: "C:Program FilesMozilla Firefox.exe" –p
   • On 64 bit : Windows: "C:Program Files(x86)Mozilla Firefox.exe" –p

4. A dialogue box will open named Firefox – choose user profile

5. Select option "Create Profile" from the window, and a wizard will open. Click on next
6. Give your profile name which you want to create and click on finish button
7. Now your profile is ready you can select your profile and open Firefox.

8. Add this code in your program :

   ProfilesIni profile = new ProfilesIni();
   
   
   FirefoxProfile myprofile = profile.getProfile("Your_Profile_Name");
   
   
   // Initialize Firefox driver
   
   
   WebDriver driver = new FirefoxDriver(myprofile);