1

Really Need help from this community!

When I am attempting to scrape the dynamic content from a travel website, the prices and related vendor info can be obtained only if I click on the "View Prices" button on the website. So I am considering using 'for loop' to click on all the 'View Prices' buttons before I do my scraping using Selenium.

The Question is that every single button can be click through browser.find_element_by_xpath().click() , but when I create a list which includes all the button info, an error pops up :

Code Block :

browser=webdriver.Chrome("C:/Users/Owner/Downloads/chromedriver_win32/chromedriver.exe")
url="https://www.cruisecritic.com/cruiseto/cruiseitineraries.cfm?port=122"
browser.get(url)
#print(browser.find_element_by_css_selector(".has-price.meta-link.show-column").text)
ButtonList=[ "//div[@id='find-a-cruise-full-results-container']/div/article/ul/li[3]/span[2]",
             "//div[@id='find-a-cruise-full-results-container']/div/article[2]/ul/li[3]/span[2]",
             "//div[@id='find-a-cruise-full-results-container']/div/article[3]/ul/li[3]/span[2]"]

for button in ButtonList:
    browser.implicitly_wait(20)
    browser.find_element_by_xpath(str(button)).click()

Error Stack Trace :

WebDriverException: unknown error: Element <span class="label hidden-xs-down" data-title="...">View All Prices</span> is not clickable at point (862, 12). Other element would receive the click: <a href="https://boards.cruisecritic.com" onclick="s_objectID='Primary Nav : Boards';">...</a>
  (Session info: chrome=63.0.3239.132)
  (Driver info: chromedriver=2.35.528161 (5b82f2d2aae0ca24b877009200ced9065a772e73),platform=Windows NT 10.0.16299 x86_64)

My question would be How can I click on all the buttons on the web page before scraping, or is there any other way to scrape the dynamic content on a webpage if we have to click on certain button to 'parse' the data into Python. The attached picture is the webpage screenshot.

Really appreciate the help from the community!

enter image description here

funie200
  • 3,688
  • 5
  • 21
  • 34
Ray
  • 71
  • 1
  • 3
  • Possible duplicate of [Selenium Web Driver & Java. Element is not clickable at point (36, 72). Other element would receive the click:](https://stackoverflow.com/questions/44912203/selenium-web-driver-java-element-is-not-clickable-at-point-36-72-other-el) – undetected Selenium Feb 13 '18 at 07:45

3 Answers3

2

You might need to go for Relative path for the Xpath you are using. It might be the case where the data which is shown is only partially present while you are performing the data.

Methods to try:

  1. Increase the wait time
  2. Change the xpath / use the relative xpath
  3. Splinter - You can use it as a browser calls regular way

You need to check for the data is it present in the DOM element while making the calls. If that's the case waiting till to load the complete page will help you out.

Shivdhwaj Pandey
  • 157
  • 1
  • 11
  • I am really new to the web scraping on dynamic content. Can you plzs give me an example! I will really appreciate that! – Ray Feb 13 '18 at 07:42
  • As I can check, Your xpath in not proper, For Example: Your First xpath select 3 elements on webpage. – Danish Absar Feb 13 '18 at 13:42
1

Hello Use the Following code to click on each price button, If you want you can also introduce a implicit wait.

for one_row_view_price in browser.find_elements_by_xpath('//span[@data-title="View All Prices"]'):
     one_row_view_price.click()

let me know if your BOT is able to click on price button

Thanks

Happy Coding

Danish Absar
  • 356
  • 4
  • 12
  • Some of the buttons are clicked( three buttons or two buttons), but that error still pops up ! How can I solve that? Thanks ! – Ray Feb 13 '18 at 17:04
1

Here is the function which is designed on the basis of your requirement:

def click_handler(xpath):
    # Find total number of element available on webpage
    xpath = re.sub('"', "'", xpath)
    total_element = browser.execute_script("""
                                                        var elements = document.evaluate("%s",
                                                        document,
                                                        null,
                                                        XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE,
                                                        null);
                                                        return elements.snapshotLength;
                                                        """ %xpath
                                                        )
    # Check if there is any element
    if(total_element):
    # Iterate over all found elements
        for element_pos in range(total_element):
            # Call click element function
            browser.execute_script("""
                             var elements = document.evaluate("%s",
                             document,
                             null,
                             XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE,
                             null);
                             var im = elements.snapshotItem(%d);
                             im.click();
                             """ %(xpath, element_pos)
                             )
            print "***" + str(element_pos + 1) + " Elements clicked"    
        print "\n****************************"
        print "Total " + str(total_element) + " Elements clicked"
        print "****************************\n"
    # Inform user that there is no element found on webpage
    else:
        print "\n****************************"
        print "Element not found on Webpage"
        print "****************************\n"
        # Element not found on Webpage

click_handler('//span[@data-title="View All Prices"]')
Danish Absar
  • 356
  • 4
  • 12