0

I want to get all the matches of the website in the code, it is the website for spaniards so change to your country's one, just select the tennis part of the website. I have seen they are in a <div> whose class name is "category-container", so with this code it should work:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


driver_path = 'chromedriver.exe'
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("--incognito")
options.add_argument('--start-maximized')
options.add_argument('--disable-extensions')

driver = webdriver.Chrome(driver_path, options=options)
driver.maximize_window()
driver.get('https://www.marathonbet.es/es/popular/Tennis+-+2398')
matches = driver.find_elements_by_xpath('//div[@class="category-container"]')

But the problem is that once it arrives to the ITF Egypt, more or less half the website, it stops recording in the variable matches the divs positions, I don't know why this happens. Does someone know what is the problem and ,if possible, a solution?

scraper
  • 59
  • 1
  • 8
  • Seems like the content is dynamically loaded. You might need to scroll down to the bottom of the page for more content to load past ITF Egypt – tbjorch Feb 11 '21 at 12:10
  • And which code would you use in order to scroll down? Because I have tried this code `driver.execute_script("window.scrollTo(0, 1080)") ` but it doesn't work. – scraper Feb 11 '21 at 12:12

1 Answers1

0

The content is dynamically loaded to that page. You need to scroll down to the bottom of the page for more content to load past ITF Egypt. Maybe this thread can be of help with scrolling.

How can I scroll a web page using selenium webdriver in python?

One way (not the prettiest, but it works) could be like this.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from time import sleep


driver_path = '/Users/tobj/.m2/repository/driver/chromedriver/chromedriver'
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("--incognito")
options.add_argument('--start-maximized')
options.add_argument('--disable-extensions')

driver = webdriver.Chrome(driver_path, options=options)
driver.maximize_window()
driver.get('https://www.marathonbet.es/es/popular/Tennis+-+2398')
footer = driver.find_element_by_class_name("footer-main")
driver.execute_script("arguments[0].scrollIntoView();", footer)
sleep(3) #waiting for content to load. Better to wait for other event 
matches = driver.find_elements_by_xpath('//div[@class="category-container"]')
tbjorch
  • 1,544
  • 1
  • 8
  • 21
  • Same I told you in the comment, don't work at least for me this code snippets at this page. – scraper Feb 11 '21 at 12:15
  • @scraper check the updated snippet for one way. However, best practice not to use sleeps like this but wait for elements to appear/disappear instead. – tbjorch Feb 11 '21 at 12:22
  • I have tried all the snippets in that page and anything workin. Did you try and ran successful code? – scraper Feb 11 '21 at 12:40
  • Yes, i ran the code in the snippet I added in my answer and it worked. have you tried adding the rows i pasted above to your example code? ill update with the entire snippet instead to show what worked. – tbjorch Feb 11 '21 at 12:43
  • Okay it works perfectly, thank you very much. Do you know an event I could choose to wait instead of sleep? – scraper Feb 11 '21 at 13:51
  • Usually what i do is to see whether there are some element e.g. a loading symbol or similar that is active during the request for data and disappears when data is loaded. In that case you can wait for such element to disappear before collecting the category container elements. – tbjorch Feb 11 '21 at 14:27
  • I got what you mean but no idea of such an item of this way... – scraper Feb 11 '21 at 14:39