selenium webdriver does not find the correct number of elements

Question

Initially, I posted my question here:

Extracting content from a dynamic web site using a Java Library

Then, after reading and applying the info from the question below:

Selenium Webdriver : not displaying the correct Li elements

I installed a selenium chrome driver (Version ChromeDriver 74.0.3729.6), my chrome browser has version 74.0.3729.169. The selenium WebDriver java object is still unable to correctly find the number of elements on my web-page, altough I simulated a scroll-down and the chrome browser that the driver opened did correctly show the total number of 20 elments.

    import java.util.List;

    import org.openqa.selenium.By;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.WebElement;
    import org.openqa.selenium.chrome.ChromeDriver;
    import org.openqa.selenium.support.ui.ExpectedConditions;
    import org.openqa.selenium.support.ui.WebDriverWait;

     public class ImmoweltBot {

    public static final String URL2 = "https://www.immowelt.at/liste/wien-2-leopoldstadt/wohnungen/mieten?sort=price&cp=2";


    public static void main(String[] args) throws Exception {
        System.setProperty("webdriver.chrome.driver", "C:\\Temp\\chromedriver.exe");

        WebDriver webDriver = new ChromeDriver();
        webDriver.get(URL2);
        WebDriverWait wait = new WebDriverWait(webDriver, 15);
        By searchResults = By.xpath("//*[contains(@class, 'listitem clear relative js-listitem')]");

        JavascriptExecutor js = (JavascriptExecutor)webDriver;
        webDriver.manage().window().maximize();
        js.executeScript("window.scrollBy(0,1000)");

        wait.until(ExpectedConditions.numberOfElementsToBeMoreThan(searchResults, 4));
        List<WebElement> elemnts = webDriver.findElements(searchResults);
        System.out.println(elemnts.size());
    }

}

My web-page:

https://www.immowelt.at/liste/wien-2-leopoldstadt/wohnungen/mieten?sort=price&cp=2

Any help will be appreciated. Thank you!

score 1 · Answer 1 · answered Jun 20 '19 at 16:18

1

It is a bit tricky.You have to use infinite loop to check the elements size() and scroll down to page once it reach 20 it will jumps out of the loop.

WebDriver driver = new ChromeDriver();
driver.get("https://www.immowelt.at/liste/wien-2-leopoldstadt/wohnungen/mieten?sort=price&cp=2");
WebDriverWait wait = new WebDriverWait(driver, 15);

while(true){

  List<WebElement> elemnts=wait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.xpath("//div[contains(@class, 'listitem clear relative js-listitem')]")));                               

   driver.findElement(By.tagName("body")).sendKeys(Keys.DOWN);

   if (elemnts.size()==20)
       {
          System.out.println(elemnts.size());   
          break;
       }               

        }

answered Jun 20 '19 at 16:18

KunduK

32,888
5
17
41

thank you! Waht shall I do on the last page though, where the maximal number of elements is 20, but the actual one could be less? – Alex Mi Jun 20 '19 at 16:33
Check count and give the condition `if(elemnts.size()>=count)` – KunduK Jun 20 '19 at 16:36
Make sure you add a timeout or max retry count for if the size never gets to 20. – Asyranok Jun 20 '19 at 18:45

score 1 · Answer 2 · answered Jun 20 '19 at 18:39

Thanks for this question, it was so challenging. So Here is my solution. This is js to smooth scroll until down.

(async function() {
function sleep() {
    return new Promise(resolve => setTimeout(resolve, 500))
};
var height;
do {
    height = document.body.scrollHeight;
    window.scrollTo({
        "behavior": "smooth",
        "left": 0,
        "top": document.body.scrollHeight
    });
    await sleep()
} while (height != document.body.scrollHeight)})();

I used async function because chomedriver.executeScript() wants async function to use 'await' statement.

String scrollWhileScrollsJS = "(async function(){function sleep(){return new Promise(resolve=>setTimeout(resolve,500))};var height;do{height=document.body.scrollHeight;window.scrollTo({\"behavior\":\"smooth\",\"left\":0,\"top\":document.body.scrollHeight});await sleep()}while(height!=document.body.scrollHeight)})();";
( (ChromeDriver) webDriver ).executeScript( scrollWhileScrollsJS );

And of course we need fluent wait. For this, i found that 'scrollY' will be equals to 'document.body.scrollHeight-innerHeight' only while we on the bottom of the page.

new FluentWait<>( webDriver ).withTimeout( Duration.ofSeconds( 10 ) )
                             .pollingEvery( Duration.ofMillis( 500 ) )
                             .until( result -> ( (ChromeDriver) webDriver ).executeScript( "return scrollY" ).equals( ( (ChromeDriver) webDriver ).executeScript( "return document.body.scrollHeight-innerHeight" ) ) );

As a result, you can use this code to scroll the page, wait for it is scrolled to the end and get elements without knowledge of how many it should be.

PS: please, do not... i mean, really, DO NOT use while(true) in your automation tests.

Any known issue while using `while(true)` in automation tests? — undetected Selenium, Jun 20 '19 at 20:04
While something goes wrong(where you 101% sure it couldn't) and you have a hung test run the day before release. — Gennady Zyablitsev, Jun 21 '19 at 06:52
Still not clear, can you add update the answer with a few lines please? — undetected Selenium, Jun 21 '19 at 06:57
Still not clear what you want to read from me? That while-true with conditional break may just not reach the condition? — Gennady Zyablitsev, Jun 21 '19 at 08:53

selenium webdriver does not find the correct number of elements

2 Answers2

Linked