Efficient way to get data using Selenium

Question

I'm writing a Selenium code using java and the functionality is as below.

1. open a browser and navigate to the given URL
2. In that URL there is a searchbox, enter the keyword polices and search.
3. There would be 10 policies listed per page and there are total 125 pages.

and then

1. Click on a link.
2. Get the data from the link.
3. Go to the previous page.
4. Click the next.
5. If done with all the policies in that page, click next and start from step 1.

Currently the issue is, sometimes it searches till 20th page and sometimes till 7th page. there is no way it is going till the end. in between it throws some error like below.

Exception in thread "main" org.openqa.selenium.WebDriverException: unknown error: Element ... is not clickable at point (358, 214). Other element would receive the click: (Session info: chrome=61.0.3163.100) (Driver info: chromedriver=2.32.498550 (9dec58e66c31bcc53a9ce3c7226f0c1c5810906a),platform=Windows NT 10.0.15063 x86_64) (WARNING: The server did not provide any stacktrace information) Command duration or timeout: 0 milliseconds Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:42:44.417Z' System info: host: 'HDC3-L-6441MKV', ip: '10.196.224.96', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_111' Driver info: org.openqa.selenium.chrome.ChromeDriver Capabilities [{mobileEmulationEnabled=false, hasTouchScreen=false, platform=XP, acceptSslCerts=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, platformName=XP, setWindowRect=true, unexpectedAlertBehaviour=, applicationCacheEnabled=false, rotatable=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.32.498550 (9dec58e66c31bcc53a9ce3c7226f0c1c5810906a), userDataDir=C:/Users/rakesh.keerthi/AppData/Local/Google/Chrome/User Data}, takesHeapSnapshot=true, pageLoadStrategy=normal, unhandledPromptBehavior=, databaseEnabled=false, handlesAlerts=true, version=61.0.3163.100, browserConnectionEnabled=false, nativeEvents=true, locationContextEnabled=true, cssSelectorsEnabled=true}] Session ID: 72dc7f70f0d15b842c217058582a10ec at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:215) at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:167) at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40) at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:82) at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:45) at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:164) at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:82) at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:646) at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:275) at org.openqa.selenium.remote.RemoteWebElement.click(RemoteWebElement.java:82) at OpenGoogleInFF.printResult(OpenGoogleInFF.java:48) at OpenGoogleInFF.main(OpenGoogleInFF.java:36)

here is my code.

import java.util.Date;
import java.util.List;

import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class OpenGoogleInFF {

    public static void main(String[] args) throws Exception {
        System.setProperty("webdriver.chrome.driver",
                "C:\\Users\\myUserName\\Downloads\\Selenium\\chromedriver.exe");
        String user = System.getProperty("user.name");
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--user-data-dir=C:/Users/" + user + "/AppData/Local/Google/Chrome/User Data");
        options.addArguments("--start-maximized");
        WebDriver driver = new ChromeDriver(options);

        driver.get("myUrl");
        Thread.sleep(6000L);
        driver.findElement(By.id("SearchText")).sendKeys("policies");
        driver.findElement(By.className("ion-ios-search")).click();
        Thread.sleep(5000L);
        int noOfPages = driver.findElements(By.xpath(".//*[@class='pagination']/li")).size() - 5;
        System.out.println("no of pages are " + noOfPages);
        System.out.println(new Date());
        for (int i = 0; i < noOfPages; i++) {
            Thread.sleep(3000L);
            List<WebElement> numberOfChildren = driver.findElements(By.xpath("//div[@ng-repeat='result in results']"));
            int count = numberOfChildren.size();
            printResult(driver, count);
            driver.findElement(By.xpath(".//*[@id='pagingControl']/ul/li[128]/a")).click();
        }
        System.out.println(new Date());
    }

    private static void printResult(WebDriver driver, int count) throws Exception {
        for (int i = 1; i <= count; i++) {
            WebElement element = (new WebDriverWait(driver, 10)).until(ExpectedConditions.visibilityOfElementLocated(
                    By.xpath("html/body/div[1]/div/div[2]/div/div/div[2]/div[2]/div[1]/div/div/div[" + i
                            + "]/ul/span[2]/div/li[1]/h3/a")));

            scrollToElementByOffset(driver, element, -200).click();
            Thread.sleep(3000L);

            (new WebDriverWait(driver, 10)).until(ExpectedConditions.presenceOfElementLocated(
                    By.xpath("html/body/div[1]/div/div[2]/div/div/section/section/div/div[4]/a[2]"))).click();
            String policyNumber = driver
                    .findElement(By
                            .xpath("html/body/div[1]/div/div[2]/div/div/section/section/div/div[6]/div[2]/table/tbody/tr[1]/td[2]"))
                    .getText();

            String policyStatement = driver.findElement(By.xpath(".//*[@id='policy-view']/section/div/div[3]/h1"))
                    .getText();

            String policyDetails = driver
                    .findElement(By
                            .xpath("html/body/div[1]/div/div[2]/div/div/section/section/div/div[8]/div/div/div[2]/div[2]/div"))
                    .getText();

            System.out.println(policyNumber + "\t" + policyStatement);
            System.out.println("\n\n" + policyDetails + "\n\n");

            driver.navigate().back();
            driver.navigate().back();

        }
    }

    private static WebElement scrollToElementByOffset(WebDriver driver, WebElement element, int offset) {
        JavascriptExecutor jse = (JavascriptExecutor) driver;
        jse.executeScript("window.scrollTo(" + element.getLocation().getX() + ","
                + (element.getLocation().getY() + offset) + ");");

        return element;
    }
}

Apologies for replacing the actual URL. this is our organization's internal site.

please let me know how can I optimize/improve my code to get the results from all the pages and also how can I cut down the total running time.

Thanks

Answer 1

A few Days Ago I work like similar type project when I click and go next page but sometimes network or browser issue I faced the same type of problem.Then I handle it by try and catch ... If throw any error I just handle it by message not terminate the program.If the same error occurs 7 at a time then I terminate loop otherwise it will run..

                 int  counts=0;

                   try {
       //Here I write a code and execute 

                        //Program END
                      counts=0;  //if my program end then I consider all the operation successfully and assign value count=0 


                }


                catch (org.openqa.selenium.WebDriverException e){

                     count++;
                    System.out.println(count);
                       if(count>=7){
                 //here if sequencially 7 time i get same error then i terminate current loop and send to next loop
                        break;
                        }

Using this way you can solve your problem..Hope it will help you..