Alternative ways for scrolling a page using Selenium or Beautiful soap?

Question

I am trying to scrape a news page (thenextweb.com) which has infinite scrolling pages.

I have written a function to scroll but it takes too much time to scroll. I had to use the time.sleep() because my internet connection is weak and it gets time to load new pages.

Here is my scroll down function, I have used the solution of this question: "https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python"

def scrolldown(urltoscroll):
    browser.get(urltoscroll)
    last_height = browser.execute_script("return document.body.scrollHeight")
    next_button = browser.find_element_by_xpath('//*[@id="channelPaginate"]')
    while True:
        
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(6)
        next_button.click()
        time.sleep(8)
        new_height = browser.execute_script("return document.body.scrollHeight")
        time.sleep(6)
        if new_height == last_height:
            break
        last_height = new_height

Is there any other way to handle those kinds of pages in an easier way?

Thank you

edit: the link that i want to scrape: "https://thenextweb.com/plugged/". I want to get article hrefs.

could you provide the exact link to the page you're looking to scrap? — Sebastien D, Jul 07 '20 at 09:20

coder420 · Answer 1 · 2020-07-07T09:13:06.283

Here's an example selenium code snippet that you could use for this type of purpose. It goes to the url for youtube search results on 'Enumerate python tutorial' and scrolls down until it finds the video with the title: 'Enumerate python tutorial(2020).'

driver.get('https://www.youtube.com/results?search_query=enumerate+python')
target = driver.find_element_by_link_text('Enumerate python tutorial(2020).')
target.location_once_scrolled_into_view

You can also apply this to your news scraping code.

score 0 · Answer 2 · answered Jul 07 '20 at 10:12

Well, it appears that the scroll down action triggers an API call which you could simulate with requests module to load each page.

Here is an example with the Latest News section:

  import requests
  from bs4 import BeautifulSoup

  ## The function which read the news by page
  def getNews(page):
      headers = {
          'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0',
          'Accept': 'text/html, */*; q=0.01',
          'Accept-Language': 'fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3',
          'X-Requested-With': 'XMLHttpRequest',
          'Connection': 'keep-alive',
          'Pragma': 'no-cache',
          'Cache-Control': 'no-cache',
      }

      params = (
          ('page', page),
          ('slug', ''),
          ('taxo', ''),
      )

      response = requests.get('https://thenextweb.com/wp-content/themes/cyberdelia/ajax/partials/grid-pager.php', headers=headers, params=params)
      return response.content
  
  ## Loop through page
  for page in range(2):
      print("Page", page)
      soup = BeautifulSoup(getNews(page))

      ## Some simple data processing
      for news in soup.find_all('li'):
          news_div  = news.find('div',{'class':'story-text'})
          #Check if the li contains the desired info
          if news_div == None: continue
          print("News headline:", news_div.find('a').text.strip())
          print("News link:", news_div.find('a').get('href'))
          print("News extract:", news_div.find('p', {'class':'story-chunk'}).text.strip())
          print("#"*10)
      print()

Output:

Page 0
##########
News headline: Can AI convincingly answer existential questions?
News link: https://thenextweb.com/neural/2020/07/06/study-tests-whether-ai-can-convincingly-answer-existential-questions/
News extract: A new study has explored whether AI can provide more attractive answers to existential questions than history's most influential ...
##########
News headline: Here are the Xbox Series X games we think Microsoft will show off on July 23
News link: https://thenextweb.com/gaming/2020/07/06/xbox-series-x-games-microsoft-show-off-july-23/
News extract: Microsoft will be showing off its first-party Xbox Series X games at the end of the month. We can guess what we might be ...
##########
News headline: Uber buys Postmates for $2.65 billion — and traders are into it
News link: https://thenextweb.com/hardfork/2020/07/06/uber-stock-postmates-buyout-acquisition-billion/
News extract: Uber's $2.65 billion Postmates all-stock acquisition comes less than a month after talks to buy rival GrubHub fell through. ...

Alternative ways for scrolling a page using Selenium or Beautiful soap?

2 Answers2