1

There is a text button that says "Show more matches", it is used to scroll down the page, because no more results come out. I would like to press it automatically, but there is something wrong with my code. Can you help me please?

IMPORTANT: It is not yet on this page, but soon there will be a second button "Show more meetings", because I scroll down and find "Show more meetings", then in a few months if I go down even further and find a second button "Show more matches". In the page there is no second button yet, but I would like to make it press that too (it's the same same button, so I don't think it's complicated).

So I would like to press 2, but also 3 the same, at the same time

P.S: The purpose of the request and the code is only for personal study reasons, so for personal didactic reasons, no profit. This question and this code is not for commercial or profit-making purposes.

enter image description here

driver.get("url")
driver.implicitly_wait(12)
wait = WebDriverWait(driver, 12)

try:
    wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "a[class^='event__more event__more--static']"))).click()   #because another element <div class="skOT skOT--ot"> obscures it

except Exception as ex:
    print('EX:', ex)

UPDATE:Before driver.get and the link, there is this

torexe_linux = os.popen('/home/xxxxx/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US') 

profile = FirefoxProfile('/home/xxxx/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False) #certi la tengono True
profile.update_preferences()

firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = '/usr/bin/firefox' 

driver = webdriver.Firefox(
    firefox_profile=profile, options=firefox_options, 
    executable_path='/usr/bin/geckodriver')
Erling Olsen
  • 1
  • 4
  • 15

1 Answers1

1
  1. You need to close accept cookies panel.
  2. Show more button appears on the bottom of the page, you need to scroll it to the view in order to click it.
  3. After clicking it you need to wait for the page loading in order to get that element again, scroll to it again and the click it again.
    As following:
from selenium.webdriver.common.action_chains import ActionChains

driver.get("link")
driver.implicitly_wait(12)
wait = WebDriverWait(driver, 12)
actions = ActionChains(driver)

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()

while(driver.find_elements_by_css_selector('a.event__more.event__more--static')):
    show_more = driver.find_element_by_css_selector('a.event__more.event__more--static')
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    actions.move_to_element(show_more).perform()
    time.sleep(0.5)
    show_more = driver.find_element_by_css_selector('a.event__more.event__more--static')
    show_more.click()
    time.sleep(3)
Erling Olsen
  • 1
  • 4
  • 15
Prophet
  • 32,350
  • 22
  • 54
  • 79
  • I thank you. I have 2 questions please: 1. What is the row then you need to click on "Show more meetings"? Is it "actions.move etc ..." to click?....................2. For the second "Show more matches" button (it is not on the page, but will be added soon) do you need another "driver.execute_script" to scroll down the second time? Do you have to insert it before show_more.click ()? Because in fact it will be necessary to scroll under everything again for a second time (the page will be very long). If so, can you show me how? (I don't want to make mistakes). Thanks – Erling Olsen Nov 11 '21 at 05:52
  • 1) `actions.move_to_element(show_more).perform()` scrolls `show_more` element into the view. 2) It's not another `show_more` element, it is the same element on the page that appears on the page in case there are more results to show. It is initially out of the view, on the bottom, so you have to scroll down in order to reach it. The code makes a loop: validates if `show_more` element exists on the page? If yes - scroll down, scroll to that element, click on it, make pause to let the page loaded, make the entire process again. – Prophet Nov 11 '21 at 07:41
  • I'm sorry, it should be a `while` there, not `if` to make this block repeatable. I updated the code. – Prophet Nov 11 '21 at 07:41
  • Don't worry about while and if. I hadn't tested yet due to lack of time. I still don't understand one thing. Divide into two questions please. 1. You said "actions.move_to_element (show_more) .perform ()" scrolls show more, but I thought it was driver.execute_script that scrolled. So "driver.execute_script ......." what is it for? what is its purpose? ..... 2. What line do you need to click on the "Mostra più incontri" button? Chievo because I don't see a click (). Thanks – Erling Olsen Nov 11 '21 at 17:23
  • `driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")` scrolls the page up for a whole current page height. With no relation to the "show more" button. While `actions.move_to_element(show_more).perform()` scrolls to the specified element. In a normal situation ``actions.move_to_element(show_more).perform()`` will be enough alone, but since here we have to scroll more that a whole screen I added both these actions. – Prophet Nov 11 '21 at 18:22
  • 1
    `show_more.click()` performs a click. Since `show_more` element can be changed due to scrolling (I have faced such things) now I added one more `show_more = driver.find_element_by_css_selector('a.event__more.event__more--static')` to get this element again. – Prophet Nov 11 '21 at 18:24
  • Ah okok, understood. So the steps in order are these? 1. Click on the cookies button and close them ..... 2. You get the item "Show more" ..... 3. The page return to the top ..... 4. The page scrolls down to "Show more" ..... 5. Wait 0.5 seconds ..... 6. You get the second "Show Top" element which is even lower .....7. Wait 3 seconds – Erling Olsen Nov 11 '21 at 18:46
  • Not exactly. 1)Close the cookies panel. 2)Check if "show more" element presents on the page. 3)If exists get that element again 4) Scroll page up whole height 5) Scroll "show more" element into the view 6) Get "show more" element again to avoid StaleElement exception 7) click "show more" button 8)Wait to allow the page load more results – Prophet Nov 11 '21 at 20:51
  • That is fine. Just tell me one last simple thing: is the code written to click on 2 different "Show more" buttons? Or does it automatically click on all the "Show more" it finds? (so on 3, 4, or 5). Thanks – Erling Olsen Nov 11 '21 at 21:20
  • It clicks on "show more" only once per each "for" loop iteration. I'm getting that element several times for the following purposes: 1) `while(driver.find_elements_by_css_selector('a.event__more.event__more--static')):` check if that element exists on the page 2) `show_more = driver.find_element_by_css_selector('a.event__more.event__more--static')` first time get the "show more" element to scroll it into the view 3) `show_more = driver.find_element_by_css_selector('a.event__more.event__more--static')` second time get that element in order to click on it to avoid StaleElement exception – Prophet Nov 11 '21 at 21:44
  • Thank you, you have been very kind. I have been using Python recently and some things were not clear to me. However,i encounter an error in the code you wrote. The page browser opens normally as per scraping, but I immediately get this error: wait.until (EC.visibility_of_element_located ((By.CSS_SELECTOR, "button # onetrust-accept-btn-handler"))). Click () File "/home/xxx/.local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until raise TimeoutException (message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: >>> – Erling Olsen Nov 11 '21 at 22:25
  • That's strange... I see this element, there is no iframe there... Maybe try using `element_to_be_clickable` instead of `visibility_of_element_located` there? – Prophet Nov 12 '21 at 09:04
  • Tried. I'm still getting the same error. What if the problem was By.CSS_SELECTOR or by_css_selector? – Erling Olsen Nov 12 '21 at 15:14
  • Can you help me please? I'm stuck. I don't understand why your code doesn't work. Thanks – Erling Olsen Nov 13 '21 at 10:44
  • 1
    Maybe you should maximize your screen / set the screen size big enough? You didn't share ALL your code so I can only guess... To set the window size add this attribute: `options.add_argument("window-size=1920,1080")`. See here or similar questions: https://stackoverflow.com/questions/55012755/how-to-set-window-size-in-selenium-chrome-python – Prophet Nov 13 '21 at 16:53
  • Should setting a size solve the problem? To the code I wrote above I only added driver.minimize_window (). I have no size. Why should joke maximization solve the problem? However I saw that I can also write options.add_argument ("- start-maximized") – Erling Olsen Nov 13 '21 at 18:11
  • In case you set the screen to maximized it should work correctly. – Prophet Nov 13 '21 at 18:43
  • I thank you. tomorrow I'll try and I'll answer you (at the latest the day after tomorrow based on my time). Don't worry, don't forget to vote your answer. Thanks – Erling Olsen Nov 13 '21 at 23:12
  • Hello. Here I am. I tried to experiment, but I couldn't. I use Firefox and not Chrome, so I have seen both from your suggested link and from here https://stackoverflow.com/questions/52769997/selenium-webdriver-firefox-options-what-is-it-about. But do I necessarily have to set the page size? Because I would like not to display the window. For example I have seen that they use options.add_argument ("- headless"). I added my other code to the question. Your response was kind, but I haven't solved the problem. Could you help me please? Thanks – Erling Olsen Nov 17 '21 at 15:30
  • 1
    In case you don't set the driver screen size it will be opened with default screen size. This may cause differences between what is normally visible for you as a user and what will be actually visible on the screen of default size. This becomes more problematic in headless mode since in headless mode the default screen size is significantly smaller than regular screen size we normally use. Also I would advise you first to make your code working in a regular mode and only after that to switch to the headless. – Prophet Nov 17 '21 at 15:41
  • Ah, ok, I understand. So for the moment no headless mode. For completeness from your answer, could you help me set a page size to my code please? Sorry if I ask you, I don't want to bother you, but this way you complete your answer and I vote for you, because even if you were very kind I didn't solve the problem with your answer. I have never set the size and I don't know how to do it. I tried somehow with your link and mine, but couldn't. Thanks :) – Erling Olsen Nov 17 '21 at 15:45
  • Try this: https://stackoverflow.com/questions/15397483/how-do-i-set-browser-width-and-height-in-selenium-webdriver and this https://stackoverflow.com/questions/55012755/how-to-set-window-size-in-selenium-chrome-python it should be enough – Prophet Nov 17 '21 at 15:50
  • 1
    As about my answer completeness - each question here should be specific. It is not about "make my project working" or "learn me all I missing to make my code working". It should be a narrow, specific, question. As you can see - I already described you much more than you initially asked about. However you had to ask each such question separately 1) to make the answers be visible and usably for many other people who will come here after you too 2) to let other people answer you as well. Asking questions in comments under old question breaks both these rules. – Prophet Nov 17 '21 at 15:57
  • Being new to Stackoverflow I have yet to learn a few things. Then your answer is complete, indeed, it is more than concrete. Thank you very much. If I don't solve the problem, I'll ask another question separately. Vote your answer and thank you again for the answer and for the comments. You have been very kind. I greet you with esteem and respect. Thanks – Erling Olsen Nov 17 '21 at 16:02
  • You are more than welcome, my friend – Prophet Nov 17 '21 at 16:31
  • For privacy and security reasons can you remove the link from your answer please? Maybe replace it with the word "link" or whatever you want. Sorry for the request. I thank you in advance. P.S: I removed the green check from the answer just to write to you, then I will put it again after you have given me the confirmation that the link has been removed, so that in the meantime we can continue writing. Thanks – Erling Olsen Nov 25 '21 at 22:22
  • 1
    I approved your edit request – Prophet Nov 25 '21 at 22:43
  • Thank you, you have been very kind :) – Erling Olsen Nov 25 '21 at 22:45