1

I am trying to scroll down a YouTube Channel's "Videos" tab so I can obtain the links to all the videos in that channel. I am using the Selenium module in Python 3, with Google Chrome as the browser. This is the method in my class that does this part (this is called right after I call self.driver.get(CHANNEL_URL))

def get_video_links(self):
    xpath = '/html/body/ytd-app/div/ytd-page-manager/ytd-browse/ytd-two-column-browse-results-renderer/div[1]/ytd-section-list-renderer/div[2]/ytd-item-section-renderer/div[3]/ytd-grid-renderer/div[1]/ytd-grid-video-renderer/div[1]/div[1]/div[1]/h3/a'
    elements = self.driver.find_elements_by_xpath(xpath)
    last_result = 0
    curr_result = len(elements)
    while curr_result != last_result:
        self.driver.find_element_by_tag_name('body').send_keys(Keys.END)
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(3)
        elements = self.driver.find_elements_by_xpath(xpath)
        last_result = curr_result
        curr_result = len(elements)
    video_links = [elem.get_attribute('href') for elem in elements]
    return video_links

This currently returns 60 urls, while doing no scrolling gets me 30 urls. The channel I'm scraping, however, has around 150 videos. I have succeeded with manual scrolling, so I know the element-finding part works. I have tried this and this but they only get me 30 urls and I don't see any scrolling happening on the browser.

rcmenoni
  • 29
  • 8
  • the `youtube-dl` utility can take the identifier of a channel, list, or individual video (probably others, user_id?), and download the whole set. Not sure if that meets your use case. Good luck. – shellter Aug 26 '20 at 01:37
  • Thanks, probably, I'll take a look! – rcmenoni Aug 26 '20 at 02:20
  • good! `youtube-dl` has a very active development group/community and support. Sorry, don't have the URL handy, but it's worth searching for. `youtube-dl` is also a well known entity here on S.O., but for good intro material and advanced problems, best to look at the external site. Good luck. – shellter Aug 26 '20 at 02:28
  • youtube-dl works, thanks! – rcmenoni Aug 26 '20 at 03:28
  • Don't try social media to automate. if you want to work with scroll page try this options in this link https://www.guru99.com/scroll-up-down-selenium-webdriver.html – Justin Lambert Aug 26 '20 at 05:55

3 Answers3

3

You can use JavaScript to scroll down in selenium.

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    Taken from the selenium docs.

If you just want to download all the videos a channel has made, youtube-dl is the better job.

All you have to do is run youtube-dl from the console and add the Channel ID for the channel you want downloaded.

$ youtube-dl "[INSERT CHANNEL-ID]"

It also works on playlists and individual videos and even supports other sites.

WyattBlue
  • 591
  • 1
  • 5
  • 21
  • I've already tried that JavaScript and it doesn't scroll for some reason, it only worked to a degree after I added `driver.find_element_by_tag_name('body').send_keys(Keys.END)` to it. I'll be using youtube-dl. Thanks. – rcmenoni Aug 26 '20 at 03:22
0

This issue may be happening because after loading 60 videos, the page needs to load more videos and because of which not able to return you the full no. of videos of channel.

You can do one thing, after 60 videos which have loaded you may again search the videos link after that.

Jaspreet Kaur
  • 119
  • 4
0

This doesn't work any longer as document.body.scrollHeight is 0 now. Should use document.getElementById('page-manager').scrollHeight instead.

Silent Sojourner
  • 146
  • 4