1

The following sections of my code work but the problem here is it only gets the names of the videos in the visible part of the page. What I want to do is to scroll down through the page. Is there a way to scroll down using requests module in python??

    def __init__(self):
    word = input("Search: ")
    self.r = requests.get('https://www.youtube.com/results?search_query={}'.format(word))
    self.soup = BeautifulSoup(self.r.content,"html.parser")

def find_video(self):
    videos = self.soup.find('div',attrs={"id":"content"}).find_all("div",attrs={"class":"yt-lockup-content"})
    for video in videos:
        user_detector = video.a.get("href")
        if user_detector.startswith("/watch"):
            print(video.a.text)
            print("------------------------------------")
        else:
            pass
Resul Bulbul
  • 45
  • 1
  • 10

2 Answers2

1

requests does not interpret JavaScript. You have to use Selenium if you want to have same behavior as your have using browser. The content on page is dynamically loaded via ajax. Therefore request is not good for this.

Jan Lipovský
  • 341
  • 2
  • 5
1

Since you are not using the offical API you can't just do that by scraping using requests/BeautifulSoup. You need to execute Javascript in order to make that happend.

My suggestion would be using a webdriver that interacts with a browser directly and is able to execute JS.

from selenium import webdriver
import time
bot = webdriver.Firefox()
url = 'https://www.youtube.com/results?search_query={}'.format(word)
bot.get(url)
#waiting for the page to load
time.sleep(3) 
#repeat scrolling 10 times
for i in range(10):
    #scroll 300 px
    bot.execute_script('window.scrollTo(0,(window.pageYOffset+300))')
    #waiting for the page to load
    time.sleep(3)
n0ra5
  • 71
  • 2