0

I'm using Selenium Webdriver to get the content of a webpage. The page loads more content as the user scrolls down using AJAX. I am able to scroll down using javascript, but don't know when to stop scrolling. How do I know when I can't scroll down anymore? I can't use document.clientheight or any of the height properties, because they deal with scrolling when the window is too small but the elements are all attached to the DOM and their height is therefore known. Here, the elements are being dynamically attached to the DOM, so the height is not known in advance.

How do I know when I've reached the bottom? Is there a "scrollbar is not scrollable" condition available somewhere? Or will that also be triggered if more content is loading and the scrollbar is not yet available to scroll, but soon will be?

Thanks, bsg

Update

The answer that worked for me was actually given to another question of mine (this one was a more precise subset of that one). Namely, use jquery to test the following equivalence: $(document).height() == ($(window).height() + $(window).scrollTop();. If it evaluates to true, you've reached the bottom of the page. (For an explanation of why this works, see Meaning of $(window).scrollTop() == $(document).height() - $(window).height()) I'm not sure if this would work without jquery, so if the page you're testing doesn't have jquery, you probably should use user1177636's answer below.

The exact code I used was:

JavascriptExecutor js = (JavascriptExecutor)driver;
do{

  //scroll down and do whatever processing you need

  reachedbottom = Boolean.parseBoolean(js.executeScript("return $(document).height() == ($(window).height() + $(window).scrollTop());").toString());

  }while(!reachedbottom);

Hope this helps someone.

Community
  • 1
  • 1
bsg
  • 825
  • 2
  • 14
  • 34
  • What's the next thing you do once it reaches the bottom? If that's doable, then stop scrolling. Would this logic suit you? – Yi Zeng Jul 12 '13 at 22:21
  • I wish it did. I am scrolling in order to get all the items on the page, and when I get down to the bottom, I want to do some processing on the items I've retrieved. Since they don't have unique ids or names, I can't wait for any specific one to appear to know if I've reached the bottom. – bsg Jul 14 '13 at 03:39
  • How about check `readyState=complete`? – Yi Zeng Jul 14 '13 at 04:28
  • I'm trying to write the code to try your suggestion, but I'm not terribly familiar with JS, unfortunately. Can you specify exactly which object readyState is a property of? From searching, there seem to be more than one. And would I put this into a WebDriver wait? How do I have it wait for the JS (if that's what it is) to be executed? – bsg Jul 15 '13 at 02:15

2 Answers2

0

I had the same problem. I've just executed $('.footer').scrollIntoView() js to scroll to bottom.

Pazonec
  • 1,549
  • 1
  • 11
  • 35
  • Thanks for your response, but this won't work for me, because I need to scroll down one screen at a time and retrieve some elements from the page. I actually got an answer for this on a different question of mine (which included this question but was more general), but I'm not going to close this one yet because I'd like to see if user1177636 will respond to my question in the comment above. – bsg Jul 15 '13 at 02:17
  • the most reliable way is to call the javascript function on your page with attaches elements when you scrolling to bottom. – Pazonec Jul 15 '13 at 13:25
0

As OP wants me to give a respond here, I'll try write something down, but may not be the correct answer for this question.

First, I need to know if your site is endless scrolling or not, like Flickr Explore.

If it's endless, then I doubt there's anything to determine when it will end. You can only check if one particular element exists (which is not your case, right?).

However, if you site will end at some point, you can keep scrolling, if readyState!=complete

As you didn't say what language you are using, I'll illustrate in C#. 

bool readyStateComplete = false;
while (!readyStateComplete) {
    IJavaScriptExecutor executor = driver as IJavaScriptExecutor;
    executor.ExecuteScript("window.scrollTo(0, document.body.offsetHeight)");
    readyStateComplete = (string)executor.ExecuteScript("return document.readyState") == "complete";
}
Yi Zeng
  • 32,020
  • 13
  • 97
  • 125
  • Thanks for the example. I'm actually using Java, but should be able to translate. My site is thankfully not endless scrolling, so this should work. The problem that I still have, though, is that if I scroll too fast, the content doesn't load quickly enough and I end up with StaleElementExceptions. (At least, that's what I assume is the cause of the error, since it doesn't occur when I step through each line.) Is there a way to wait for readyState to be complete? Would that work? Should I start a new question about this particular issue? – bsg Jul 15 '13 at 03:08
  • Where is `StaleElementException` thrown? I don't think `JavaScriptExecutor` cares about that as long as no elements involved. – Yi Zeng Jul 15 '13 at 03:10
  • My original answer was waiting for readyState to be complete, then realize it won't work. Because it will be completed even if it's not at the bottom, then you scroll once more, page loads, then readyState=complete, then you can still scroll down again. – Yi Zeng Jul 15 '13 at 03:12
  • I'm sorry - I'm not being clear. I had a few issues, which I put into a single question, and then I put this one in a separate question. You answered this question already, but I was hoping that your original answer will help me for another issue I'm having. (Namely, that my content doesn't load fast enough, so trying to access it before it finishes loading causes errors. I need some way to wait for it to finish loading.) – bsg Jul 15 '13 at 12:34
  • I'm going to accept your answer because it should work, but I actually used an answer that I got to another question, which I'll add here. Thanks! – bsg Jul 15 '13 at 12:57