0

Context

This is a repost of Get a page with Selenium but wait for element value to not be empty, which was Closed without any validity so far as I can tell.

The linked answers in the closure reasoning both rely on knowing what the expected text value will be. In each answer, it explicitly shows the expected text hardcoded into the WebDriverWait call. Furthermore, neither of the linked answers even remotely touch upon the final part of my question:

[whether the expected conditions] come before or after the page Get

"Duplicate" Questions

Original Question

I'm grabbing a web page using Selenium, but I need to wait for a certain value to load. I don't know what the value will be, only what element it will be present in.

It seems that using the expected condition text_to_be_present_in_element_value or text_to_be_present_in_element is the most likely way forward, but I'm having difficulty finding any actual documentation on how to use these and I don't know if they come before or after the page Get:

webdriver.get(url)

Rephrase

How do I get a page using Selenium but wait for an unknown text value to populate an element's text or value before continuing?

muad-dweeb
  • 995
  • 1
  • 9
  • 24
  • Can you add the HTML before and after the element is present? Did you have a look at https://stackoverflow.com/questions/28240342/perform-a-webdriverwait-or-similar-check-on-a-regular-expression-in-python? – Maximilian Peters Nov 23 '20 at 21:17
  • @MaximilianPeters The particular element I need seems to always be present in the scraped document, it just doesn't have a text value in it. When I load the page in a browser, the value is blank for a second or two before it populates. I'd love to post the raw HTML, but I'm now dealing with an auth issue preventing me from logging into the site. I'll update with raw HTML once the client fixes their auth. Your linked question looks promising. I'll give it a shot once I'm unblocked. – muad-dweeb Nov 23 '20 at 21:43
  • seems like you just need a polling loop here... check to see if the element is empty. If it's not, then continue. You don't need a webdriverwait or an expected condition for this. – pcalkins Nov 23 '20 at 22:51
  • @pcalkins I guess I was under the impression that once the page is grabbed with `driver.get(url)` that it exists as a static object, but if it will continue to load its elements then my problem should be effectively solved. I'm eager to try it out once I regain access... – muad-dweeb Nov 24 '20 at 01:06
  • sounds like a "lazy-loading" site which fires a script to populate the DOM after the page fully loads. – pcalkins Nov 24 '20 at 17:32

1 Answers1

0

I'm sure that my answer is not the best one but, here is a part of my own code, which helped me with similar to your question.

In my case I had trouble with loading time of the DOM. Sometimes it took 5 sec sometimes 1 sec and so on.

url = 'www.somesite.com'
browser.get(url)

Because in my case browser.implicitly_wait(7) was not enought. I made a simple for loop to check if the content is loaded.

some code...

 for try_html in range(7):
        """ Make 7 tries to check if the element is loaded """
        browser.implicitly_wait(7)
        html = browser.page_source
        soup = BeautifulSoup(html, 'lxml')
        raw_data = soup.find_all('script', type='application/ld+json')
        

       """if SKU in not found in the html page we skip 
         for another loop, else we break the 
          tryes and scrape the page"""

        if 'sku' not in html:
            continue
        else:
            scrape(raw_data)
            break

It's not perfect, but you can try it.

BBorisov95
  • 41
  • 9
  • 1
    Thanks @BBorisov95. I'll hold onto this one as a last resort. I want to avoid hardcoding a wait for every page, because that will slow down my scrape by a fixed amount, whereas if I can tell it to wait for a certain value to load, then theoretically it should dynamically wait for only as long as is necessary for each page. – muad-dweeb Nov 23 '20 at 21:22