Obtaining the number of comments of a list of youtube videos

Question

I have been coding a simple python script for obtaining number of views and number of comments of a list of videos. Using csv, I have converted a tab-separated table into a list of lists, and then I tried to obtain both elements. Inspecting number of views, the element is "div", {"class":"watch-view-count"}. It works as intended

r = requests.get(list_youtube_reading[n][0]) # it retrieves each video URL from a csv
soup = BeautifulSoup(r.text)
for element in soup.findAll("div", {"class":"watch-view-count"}): 
    patternviews = re.compile('^(.*?) .*') 
    scissorviews = patternviews.match(element.text.encode("utf-8")) 
    views = re.sub('\.','', tijeraviews.group(1))

However, element for number of comments is <h2 class="comment-section-header-renderer" tabindex="0"> <b>Comments</b> " • 6" <span class="alternate-content-link"></span>
</h2>

When I tried to obtain it, with

for element in soup.findAll("h2", {"class":"comment-section-header-renderer"}):
    comments = element.text.encode("utf-8")
    print comments

nothing happens, and actually soupdoesn't contain any <h2 class="comment-section-header-renderer" tabindex="0"> tag

What can I do in order to retrieve number of comments? I tried to use youtube v3 data API, but for no avail

thanks in advance

Answer 1

One simple way would be using the Selenium WebDriver to simulate a web browser. I have observed that when we scroll down, only then YouTube loads the comments section. So my solution is to make the web-driver to scroll down and wait until the desired element is found. After it has been located, the following script grabs it and gets the value.

For using Selenium, we need to download one of the third party drivers from this page. I have used the Mozilla GeckoDriver. And we also need to put the path to this executable file in the system environment variables. As I am on an Ubuntu machine, I put the downloaded file (after extracting it) in /usr/local/bin/and I didn't need anything more. After setting the path properly, we can run the following script to get our desired values. And we also do need to install Selenium itself. The instructions are here.

# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

video_url = 'https://www.youtube.com/watch?v=NP189MPfR7Q'
driver = webdriver.Firefox()
driver.set_page_load_timeout(30)
driver.get(video_url)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
for view_num in driver.find_elements_by_class_name("watch-view-count"):
    print 'Number of views: ' + view_num.text.replace(' views', '')

try:
    element = WebDriverWait(driver, 30).until(
        EC.presence_of_element_located((By.CLASS_NAME, "comment-section-header-renderer")))
    for comment_num in driver.find_elements_by_class_name("comment-section-header-renderer"):
        print u'Number of comments: ' + comment_num.text.replace(u'COMMENTS • ', '')
finally:
    driver.quit()

Output:

Number of views: 3,555
Number of comments: 3

NOTE Since the DOM element (that contains the comment-count) has some NON-ASCII character inside, I needed to put the very first line of the script.

And if you don't like Selenium to show the GUI, follow these instructions. I did not do this but the instructions should be enough.

Answer 2

It appears the comments section is loaded in a separate ajax request to a URL like this:

https://www.youtube.com/watch_fragments2_ajax?v=zlYDDLCorNw&tr=time&distiller=1&ctoken=EhYSC3psWURETENvck53wAEAyAEA4AEBGAY%253D&frags=comments&spf=load

That returns some json like this:

{
  "name": "other",
  "foot": "<script>...</script>",
  "body": {
    "watch-discussion": " ... <h2 class=\"comment-section-header-renderer\" tabindex=\"0\">\n<b>Comments</b> • 2<span class=\"alternate-content-link\"> ..."
  }
}

In that json is where you'll find the HTML section showing the comment count (in body.watch-discussion).