0

I am currently trying to scrape data from the website skiplagged.com. The webpage that I am currently trying to scrape data of has a ton of information which is beneficial for me.

For example, let us say that there is information regarding 100 flights on the webpage. Despite of targeting the right element to get the information from, I only managed to obtain information related to 10 flights only.

I later realized that new divs were getting added as I kept on scrolling down the webpage. As a result, I am not able to scrape the remaining portion of the webpage.

Earlier Attempts

I also visited this link to make an attempt to go at the end of the page to finally scrape the data: How to scroll to the end of the page using selenium in Python?

However, my attempts yielded only failures.

Link to the webpage I am trying to scrape data from

https://skiplagged.com/flights/YTO/DXB/2020-08-21

My Python Code

infinite_list = driver.find_elements_by_xpath("//div[@class='infinite-trip-list']//div[@class='span1 trip-duration']")
for elem in infinite_list:
    print(elem.text)

1 Answers1

0

You can use execute_script for this with a little tweak

time.sleep(3) #sleeping to let the page load
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") #scrolling down to the end

You can also do infinite scrolling until the page ends

last = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(1) #let the page load
    new = driver.execute_script("return document.body.scrollHeight")
    if new == last: #if new height is equal to last height then we have reached the end of the page so end the while loop
        break
    last = new #changing last's value to new
Just for fun
  • 4,102
  • 1
  • 5
  • 11