I've written a script in python with selenium to scrape the complete flight schedule from a webpage. Upon running my script I could see that it is working good so far except for some fields which are not getting parsed. I've checked for the elements within which the data are located but I noticed that elements for already scraped one and the missing one are no different. What to do to get the full content. Thanks in advance.
Here is the script I'm trying with:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("http://www.yvr.ca/en/passengers/flights/departing-flights")
wait = WebDriverWait(driver, 10)
item = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.yvr-flights__table")))
list_of_data = [[item.text for item in data.find_elements_by_css_selector('td')]
for data in item.find_elements_by_css_selector('tr')]
for tab_data in list_of_data:
print(tab_data)
driver.quit()
Here is the partial picture of the data [missing one and scraped one]: https://www.dropbox.com/s/xaqeiq97b6upj5j/flight_stuff.jpg?dl=0
Here are the td elements for one block:
<tr class="yvr-flights__row yvr-flights__row--departed " id="226792377">
<td>
<time class="yvr-flights__label yvr-flights__scheduled-label yvr-flights__scheduled-label--departed notranslate" datetime="2017-08-24T06:20:00-07:00">
06:20
</time>
</td>
<td class="yvr-flights__table-cell--revised notranslate">
<time class="yvr-flights__label yvr-flights__revised-label yvr-flights__revised-label--departed" datetime="2017-08-24T06:20:00-07:00">
06:19
</time>
</td>
<td class="yvr-table__cell yvr-flights__flightNumber notranslate">AC560</td>
<td class="hidden-until--md yvr-table__cell yvr-table__cell--fade-out yvr-table__cell--nowrap notranslate">Air Canada</td>
<td class="yvr-table__cell yvr-table__cell--fade-out yvr-table__cell--nowrap notranslate">San Francisco</td>
<td class="hidden-until--md yvr-table__cell yvr-table__cell--nowrap notranslate">
Main
</td>
<td class="hidden-until--md yvr-table__cell yvr-table__cell--nowrap notranslate">E87</td>
<td class="yvr-flights__table-cell--status yvr-table__cell--nowrap">
<span class="yvr-flights__status yvr-flights__status--departed">Departed</span>
</td>
<td class="hidden-until--md yvr-table__cell yvr-table__cell--nowrap">
</td>
<td class="visible-until--md">
<button class="yvr-flights__toggle-flight">Toggle flight</button>
</td>
</tr>
