9

I have a web page which loads content dynamically and while page loads, there is spinning wheel, I already found solution to grab content loaded immediately on page, but seems i can't find solution to grab content loaded later in dom.

What i can think of is to find element with specific class of that wheel spinning, and wait for it to change, once it's changed, than it means content is loaded in dom.

I am using Selenium with Firefox webdriver on Ubuntu.

Here is the class i am looking to monitor:

<div class="wheel spinning"></div>

Once content is loaded, wheel stop spinning and class is changed to:

<div class="wheel"></div>

Anyone find solution to find and monitor class="wheel spinning" and once it's changed to class="wheel" to continue to grab data.

Edit:

The XPATH actually solved one part of solution, here's part of code

try:
    element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "//*[@class='wheel']))
)
title = driver.find_element_by_xpath('/html/body/div[1]/div[1]/div[3]')
print(title.text)

But if element don't appear within 10 seconds it error's out, now to find a way to retry again and again until element is present on page.

Is there a difference in use presence_of_element_located((By.XPATH)) and find_element_by_xpath

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
lonerunner
  • 1,282
  • 6
  • 31
  • 70
  • 1
    I think you might be able to use a variant of what's here: https://stackoverflow.com/a/26567563/142780, specifically where they wait for a particular item (in that case found via an id rather than a class): WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement'))) – Neil Nov 10 '18 at 15:03
  • I tried this, but problem is ID always change on page, every time page refresh id of element get different, while class stays the same. I am looking also at css_selector options and xpath but none of them working for my tests. – lonerunner Nov 10 '18 at 15:07
  • 1
    But isn't there a By.CLASS_NAME? I think I'd best leave it to you, but I'm sure it's possible as did this kind of thing myself (just was a long time ago and don't have code to hand to give you the exact right syntax). NB: if you're using XPATH there are lots of simple slip ups it's easy to make, so it can be handy to test your XPATH in the Console first – Neil Nov 10 '18 at 15:16
  • CLASS_NAME only accept one class, it can't find 'wheel spinning` only 'wheel' but i don't know if it will change in 10 or 40 seconds to that class. How i test it is actually i call terminal from php script and call python script on that. – lonerunner Nov 10 '18 at 15:18
  • 1
    I've just seen your XPATH edit. Does WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.XPATH, "//*[@class='wheel']))) not work? Or is it matching when both classes are present? (ie wheel and spinning). I'm thinking you can skip monitoring the 'wheel spinning' version and simply wait until the appearance of a class with just 'wheel'. – Neil Nov 10 '18 at 15:58
  • 1
    You are right, this really do the trick. it wasn't working with CLASS_NAME or CSS_SELECTOR but XPATH looks like it's looking for exact class. – lonerunner Nov 10 '18 at 16:23

2 Answers2

8

You can wait for the class value to change. For example:

from selenium.webdriver.support.ui import WebDriverWait

# Wait longer than 10 seconds since you're getting occasional timeout
el = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "//*[@class='wheel']")))


wait = WebDriverWait(driver, 10)
wait.until(lambda d: 'spinning' not in el.get_attribute('class'))

The until method passes the driver to the method given, so you can make your own expected condition pretty easily. The above uses an anonymous lambda function but you could also use a closure or a anything callable that takes in an argument (the ExpectedConditions library is just a set of callable classes). Here is the same with a closure:

from selenium.webdriver.support.ui import WebDriverWait


# Wait longer than 10 seconds since you're getting occasional timeout
el = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "//*[@class='wheel']")))

def wait_not_spinning(driver):
    return 'spinning' not in el.get_attribute('class')

wait = WebDriverWait(driver, 10)
wait.until(wait_not_spinning)
Lucas Tierney
  • 2,503
  • 15
  • 13
  • Can you explain, what is meaning behind lambda d: ? Thanks – lonerunner Nov 11 '18 at 13:45
  • Lambda is just an anonymous function. The `until` automatically calls the passed in driver to the method given so that's what the `d` argument for the lambda is. I'll update it to show the same with a closure – Lucas Tierney Nov 11 '18 at 15:43
6

@LucasTierney's answer was in the right direction. However I still feel the solution can be optimized as follows:

As the wheel is visible, instead of presence_of_element_located() method you need to use visibility_of_element_located() method.

The node:

<div class="wheel spinning"></div>

Can't be located through the XPath containing a single class i.e. only wheel as in:

el = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "//*[@class='wheel']")))

Instead you can use either of the Locator Strategies:

  • cssSelector:

    el = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.wheel.spinning")))
WebDriverWait(driver, 10).until(lambda d: 'spinning' not in el.get_attribute('class'))

  • xpath:

    el = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='wheel spinning']")))
WebDriverWait(driver, 10).until(lambda d: 'spinning' not in el.get_attribute('class'))
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352