0

I'm trying to parse the text of all span that are inside the but when I call 'soup_dc.find_all('div', {'class': 'box-row-event'})' it only finds me 3 div and not all (they are supposed to be 7)

 from selenium import webdriver
 from bs4 import BeautifulSoup
 import time

 #Apri la pagina
 driver = webdriver.Firefox(executable_path='/Applications/Python 3.7/geckodriver')
 driver.get('https://www.eurobet.it/it/scommesse/?splash=false#!/calcio/se-superettan/')
 time.sleep(3)

 # Doppia chance
 dc_button = driver.find_element_by_link_text('doppia chance')
 dc_button.click()
 time.sleep(15)

 # Page source
 source_dc = driver.page_source
 soup_dc = BeautifulSoup(source_dc, 'lxml')

 event_container = soup_dc.find_all('div', {'class': 'box-row-event'})
 print(len(event_container)) #it shows only 3 and not 7

 for event in event_container:
     for div in event.find_all('div', {'class': 'ctn box-wrap'}):
         for dc in div.find_all('span'):
             print(dc.text)
Antonio Spatuzzi
  • 41
  • 1
  • 9
  • you need a slow scroll to generate the other items – QHarr Jul 08 '19 at 17:49
  • Thank you very much for your quick answer @QHarr. How can I do that? – Antonio Spatuzzi Jul 08 '19 at 17:52
  • For slow scrolling you can see this answer https://stackoverflow.com/questions/30942041/slow-scrolling-down-the-page-using-selenium – Ankit Agrawal Jul 08 '19 at 18:22
  • Thank you very much. I appreciate a lot your help. – Antonio Spatuzzi Jul 08 '19 at 19:09
  • Did you get it working?@AntonioSpatuzzi – Ankit Agrawal Jul 09 '19 at 19:17
  • I added this line of code: `driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")` and it actually goes till the end of the page but it still finds only the first 3 rows like before. is the line of code I added not right for my purpose? @AnkitAgrawal – Antonio Spatuzzi Jul 12 '19 at 11:47
  • with this line of code instead: `results = driver.find_elements_by_css_selector("div.box-sport") print("Results count: %d" % len(results)) driver.execute_script("arguments[0].scrollIntoView();", results[-1])` it goes to the last item and parse it but it doesn't parse all the items in between. @AnkitAgrawal – Antonio Spatuzzi Jul 12 '19 at 20:07
  • @AnkitAgrawal I used the link you shared with me to find a solution, but how can I stop a while true loop? The page keep scrolling down and never stops. – Antonio Spatuzzi Aug 25 '19 at 21:58

0 Answers0