Is there a way I can scrape this web page with selenium?

Question

So I want to scrape a list of Confirmed Participate from this site but it's returning an empty list/string

Here is my code:

driver.find_element_by_xpath('//*[@id="main"]/div/section/section[8]/section/div[1]/div[1]/a').text

Please how can I resolve this?

score 0 · Answer 1 · answered Apr 05 '21 at 23:30

Credits here : How to get text of an element in Selenium WebDriver, without including child element text?

And here for the lazy load: How to get all the data from a webpage manipulating lazy-loading method?

Your xpath isnt pointing to the right html tag

from selenium import webdriver
import time

driver = webdriver.Chrome(executable_path='D:/chromedriver.exe')


driver.get('https://www.griclub.org/event/real-estate/gri-proptech-esummit_2191.html')


def get_text_excluding_children(driver, element): 
    return driver.execute_script(""" return jQuery(arguments[0]).contents().filter(function() { 
    return this.nodeType == Node.TEXT_NODE; }).text();
    """, element)

def scroll_at_the_end_of_the_page_lazy_load():
    check_height = driver.execute_script("return document.body.scrollHeight;") 
    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(2)
        height = driver.execute_script("return document.body.scrollHeight;") 
        if height == check_height: 
            break 
        check_height = height

#Scroll untill all participants are on the page
scroll_at_the_end_of_the_page_lazy_load()

# get all CONFIRMED PARTICIPANTS
all_confirmed_participants = driver.find_elements_by_xpath("//section[@data-title ='CONFIRMED PARTICIPANTS']/descendant::div[(@class='crm-people-round-plus-text')]")
time.sleep(5)
for element in all_confirmed_participants:
    #print (element.get_attribute("innerHtml"))
    print( get_text_excluding_children(driver, element))

Is there a way I can scrape this web page with selenium?

1 Answers1