1

I have been trying to collect a list of live channels/viewers on Youtube Gaming. I am using selenium with Python to force the website to scroll down the page so it loads more that 11 channels. For reference, this is the webpage I am working on.

I have found the location of the data I want, but I am struggling with getting selenium to go there. The part I am having trouble with looks like this:

<div class="style-scope ytg-gaming-video-renderer" id="video-metadata"><span class="title ellipsis-2 style-scope ytg-gaming-video-renderer"><ytg-nav-endpoint class="style-scope ytg-gaming-video-renderer x-scope ytg-nav-endpoint-2"><a href="/watch?v=FFKSD1HHrdA" tabindex="0" class="style-scope ytg-nav-endpoint" target="_blank">
              Live met Bo3
            </a></ytg-nav-endpoint></span>
    <div class="channel-info small layout horizontal center style-scope ytg-gaming-video-renderer">
        <ytg-owner-badges class="style-scope ytg-gaming-video-renderer x-scope ytg-owner-badges-0">
            <template class="style-scope ytg-owner-badges" is="dom-repeat"></template>
        </ytg-owner-badges>
        <ytg-formatted-string class="style-scope ytg-gaming-video-renderer">
            <ytg-nav-endpoint class="style-scope ytg-formatted-string x-scope ytg-nav-endpoint-2"><a href="/channel/UCD8Q9V5wgo8o0XGfUqsRrDQ" tabindex="0" class="style-scope ytg-nav-endpoint" target="_blank">Rico Eeman</a>
            </ytg-nav-endpoint>
        </ytg-formatted-string>
    </div><span class="ellipsis-1 small style-scope ytg-gaming-video-renderer" id="video-viewership-info" hidden=""></span>
    <div id="metadata-badges" class="small style-scope ytg-gaming-video-renderer">
        <ytg-live-badge-renderer class="style-scope ytg-gaming-video-renderer x-scope ytg-live-badge-renderer-1">
            <template class="style-scope ytg-live-badge-renderer" is="dom-if"></template>

            <span aria-label="" class="text layout horizontal center style-scope ytg-live-badge-renderer">4 watching</span>
            <template class="style-scope ytg-live-badge-renderer" is="dom-if"></template>
        </ytg-live-badge-renderer>
    </div>
</div>

Currently, I am trying:

#This part works fine. I can use the unique ID
meta_data = driver.find_element_by_id('video-metadata')

#This part is also fine. Once again, it has an ID.
viewers = meta_data.find_element_by_id('metadata-badges')
print(viewers.text)

However, I am have been having trouble getting to the channel name (in this example 'Rico Eeman', and it is under the first nested div tag). Because its a compound class name, I cannot find the element by class name, and trying the following xpaths doesnt work:

name = meta_data.find_element_by_xpath('/div[@class="channel-info small layout horizontal center style-scope ytg-gaming-video-renderer"]/ytg-formatted-string'

name = meta_data.find_element_by_xpath('/div[1])

They both raise the element not found error. I am not really sure what to do here. Does anyone have a working solution?

Padraic Cunningham
  • 176,452
  • 29
  • 245
  • 321
Pieter
  • 127
  • 1
  • 10

2 Answers2

1

The name id not in the <ytg-formatted-string> tag, its in one of it descendants. Try

meta_data.find_element_by_css_selector('.style-scope.ytg-formatted-string.x-scope.ytg-nav-endpoint-2 > a')

Or with xpath

meta_data.find_element_by_xpath('//ytg-nav-endpoint[@class="style-scope ytg-formatted-string x-scope ytg-nav-endpoint-2"]/a')
Guy
  • 46,488
  • 10
  • 44
  • 88
  • Thank you, with css selector worked perfectly! However, if I print the result of the xpath with result.text it prints empty strings. EDIT: Which is no problem ofcourse, as it works with the css selector! :) – Pieter Apr 25 '16 at 09:14
0

This will get all the names, even if your xpath worked using video-metadata would not get all the names, the id is repeated per div for each user so you would need find_elements and to iterate over the returned elements:

names = dr.find_elements_by_css_selector("a.style-scope.ytg-nav-endpoint[href^='/channel/']")
print([name.get_attribute("text") for name in names])

Which gives you:

['NinjaNation Gaming', 'DURX DANIEL', 'DEMON', 'Perfection', 'The one and only jd', 'Violator Games', 'KingLuii718', 'NinjaNation Gaming', 'DURX DANIEL', 'DEMON', 'Perfection']
Padraic Cunningham
  • 176,452
  • 29
  • 245
  • 321