Append data scraped from different webpages into an arraylist

Question

I'm working on a project which involves scraping data from websites and I've run into a problem. I want to scrape data from one webpage, append it into an arraylist then have the crawler move to the next page and do the same and so on and so forth but when I execute my code the preceding data appended is overwritten and in the end I only have scraped data from the final webpage scraped. I am also encountering an error: Exception in thread "main" org.openqa.selenium.StaleElementReferenceException: stale element reference: element is not attached to the page document (Session info: chrome=71.0.3578.98) (Driver info: chromedriver=2.42.591059 (a3d9684d10d61aa0c45f6723b327283be1ebaad8),platform=Mac OS X 10.13.6 x86_64) (WARNING: The server did not provide any stacktrace information) Command duration or timeout: 0 milliseconds

Below is my code and log. Please help. Thank you. The following scrapes data and appends it to an array list:

do {

        row = (ArrayList<WebElement>) driver.findElements(By.cssSelector(".event-row-container.ng-scope"));

        WebElement element = driver.findElement(By.cssSelector(".paybillnumbers"));
        ((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", element);
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }


        new WebDriverWait(driver,20).until(ExpectedConditions.elementToBeClickable(By.cssSelector("li.ng-scope[translate='next']"))).click();
        times++;
        rows.addAll(row);
        }
        while(times <=6);

Here is the code the log refers the error to:

                ArrayList<WebElement> rowDetails2 = (ArrayList<WebElement>) rowDetails.findElements(By.cssSelector(".event-market.market-3-way.market-selections-3"));

And here is my log:

Starting ChromeDriver 2.42.591059 (a3d9684d10d61aa0c45f6723b327283be1ebaad8) on port 3035 Only local connections are allowed. Dec 18, 2018 11:18:53 AM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: OSS Starting ChromeDriver 2.42.591059 (a3d9684d10d61aa0c45f6723b327283be1ebaad8) on port 24080 Only local connections are allowed. Dec 18, 2018 11:18:58 AM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: OSS 98 Exception in thread "main" org.openqa.selenium.StaleElementReferenceException: stale element reference: element is not attached to the page document (Session info: chrome=71.0.3578.98) (Driver info: chromedriver=2.42.591059 (a3d9684d10d61aa0c45f6723b327283be1ebaad8),platform=Mac OS X 10.13.6 x86_64) (WARNING: The server did not provide any stacktrace information) Command duration or timeout: 0 milliseconds

score 0 · Accepted Answer · answered Dec 18 '18 at 09:32

For StaleElementReferenceException we need to use waits

    new WebDriverWait(driver, 60).until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.cssSelector(".event-market.market-3-way.market-selections-3")));

here 60 is max seconds, you can change. also locator contains number 3 there may be changes of changing locator on executions, so please look into it.

If waits fails you can use Thread.sleep in java also.

refer this

Append data scraped from different webpages into an arraylist

1 Answers1