0

I am in a university project building a cralwer for webpages. Now I encountered testing dropdown lists in webpage. Specifically, the following page does not use the standard "dropdown" class. 

https://www.mirrorfiction.com/zh-Hant/book/406

I have difficulties implementing a decision procedure to tell whether there are dropdowns in the webpage and whether the dropdowns have been dropped down.

The programmers seem to use a class called btnGa or something alike. I cannot google that.
I have no clue how a browser would know the dropdown tags.
The html source is too big. So I only copied a small part in the following.
The html source contains a ul of navlist class. Then an li tag in this ul contains another ul which is actually a dropdown list at hovering.
But I really cannot see how the browser would know that this is a dropdown list. Thus I don't know how to write a test case to click the buttons in this dropdown list.

Can any one help me out in recognizing the dropdown lists and click the dropdown list items ?

HTML:

<ul class="navList">
    <li class="mobileDisable">
        <a data-ga-label="小說" data-ga-action="click" data-ga-category="header" onclick="app.navLiClick(this)" class="hasSub btnGa open" href="https://www.mirrorfiction.com/zh-Hant/list/novel">
            <span class="text novel">小說</span>
        </a>
        <ul>
            <li>
                <a data-ga-label="小說-仙俠" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/11">
                        <span class="text">仙俠</span>
                        <span class="num">/ 16</span>
                    </a>
                </li>
                <li>
                    <a data-ga-label="小說-玄幻" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/9">
                        <span class="text">玄幻</span>
                        <span class="num">/ 24</span>
                    </a>
                </li>
                <li>
                    <a data-ga-label="小說-奇幻" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/29">
                        <span class="text">奇幻</span>
                        <span class="num">/ 56</span>
                     </a>
                                </li>
                                                            <li>
                     <a data-ga-label="小說-武俠" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/10">
                         <span class="text">武俠</span>
                         <span class="num">/ 11</span>
                      </a>
                  </li>
                                                            <li>
                  <a data-ga-label="小說-科幻" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/8">
                       <span class="text">科幻</span>
                       <span class="num">/ 17</span>
                  </a>
              </li>
                                                            <li>
              <a data-ga-label="小說-恐怖" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/30">
                       <span class="text">恐怖</span>
                       <span class="num">/ 11</span>
              </a>
            </li>
                                                            <li>
           <a data-ga-label="小說-校園" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/7">
                  <span class="text">校園</span>
                  <span class="num">/ 16</span>
                </a>
            </li>
                                                            <li>
            <a data-ga-label="小說-愛情" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/1">
                  <span class="text">愛情</span>
                  <span class="num">/ 78</span>
               </a>
            </li>
                                                            <li>
            <a data-ga-label="小說-寫實" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/5">
                  <span class="text">寫實</span>
                  <span class="num">/ 48</span>
                </a>
            </li>
                                                            <li>
                                    <a data-ga-label="小說-歷史" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/6">
                                        <span class="text">歷史</span>
                                        <span class="num">/ 13</span>
                                    </a>
                                </li>
                                                            <li>
                                    <a data-ga-label="小說-驚悚" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/31">
                                        <span class="text">驚悚</span>
                                        <span class="num">/ 24</span>
                                    </a>
                                </li>
                                                            <li>
                                    <a data-ga-label="小說-其它" data-ga-action="click" data-ga-category="header" class="btnGa" href="https://www.mirrorfiction.com/zh-Hant/list/novel/classification/12">
                                        <span class="text">其它</span>
                                        <span class="num">/ 40</span>
                                    </a>
                                </li>
                                                    </ul>
                    </li>
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Farn Wang
  • 143
  • 15
  • 1
    for some reason the menu doesn't show on hover in my browser but have you tried `find_elements_by_tag_name` ? Are you trying to click the links (`` tags) ? – Tom May 28 '17 at 08:03
  • Thanks ! Do you mean my work place ? or my program for testing ? I am considering how to write a program that can automatically click web pages like a monkey. But if you use the http link, can you see the dropdowns on hover? – Farn Wang May 28 '17 at 11:28
  • Thanks ! Do you mean my work place ? or my program for testing ? I am considering how to write a program that can automatically click web pages like a monkey. But I have not written it yet, since I am not sure what features to recognize the dropdown lists. If you use the http link, can you see the dropdowns on hover? – Farn Wang May 28 '17 at 11:35
  • 2
    he means ur code and ur attempt to solve this problem. It's a lot easier to help you if you pasted what you have so far and where you got stuck, also error messages, etc. – Tom May 28 '17 at 14:51
  • 1
    are you using Java? – santhosh kumar May 28 '17 at 16:58
  • Thanks, I am using python. I will try to fix the code and paste it here. At the moment, it is like a catch 22. I trying to get the features for dropdowns so that I can write the code. But then people said that I need show them the code first. – Farn Wang May 29 '17 at 01:09
  • Sorry that recently I left the problem to a grad student. If he got some nice automated solution, I will be happy to let people know. – Farn Wang Feb 23 '19 at 09:51

1 Answers1

2

Consider following the steps & lines of code to open the url & click on a menu:

  • Install current version of selenium through pip
  • Download the latest chromedriver.exe and provide the absolute path in your script

  • Code Block: 

    from selenium import webdriver
driver=webdriver.Chrome("C:\\Utility\\your_directory\\chromedriver.exe")
#maximize the browser window
driver.maximize_window()
#open the url in the browser
driver.get("https://www.mirrorfiction.com/zh-Hant/book/406")
#click on the first menu item 小說
driver.find_element_by_xpath("//nav[@id='nav']/div/ul/li/a/span[@class='text novel']").click()
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Thanks for the help. But that is the solution for manual testing and not what I was asking. I am in a university project for building a web crawler that should systematically click all the clickables. But the problem here is that I did not see how the carwler can algorithmically decide whether a dropdown clickable has already been dropped down. I downloaded the html before and after the dropping-down, it looks lie the html codes are pretty much the same. Although I am still comparing the difference manually. – Farn Wang May 29 '17 at 06:32
  • 1
    @FarnWang Let me know your exact steps so that we can help you. You Question previously was missing the `carwler` word. Thanks – undetected Selenium May 29 '17 at 08:59
  • 1
    Thanks for the comments and kindness in willingness to help. But it seems that my question is more or less research oriented. I was trying to see whether some colleagues already have a good solution so that we do not have to invent the wheel again. – Farn Wang Feb 23 '19 at 09:52