1

so when I try to find the element of a clip on twitch, it can't be found even though I am 99% sure it's right. I'm using Xpath and trying to plus one every time it loops and fetch the link to a variable to use later, but it doesn't work.

Link: https://www.twitch.tv/xqc/videos?filter=clips&range=7d

for x in range(1, antall+1):
    print("/html/body/div[1]/div/div[2]/div/main/div[1]/div[3]/div/div/div[1]/div[1]/div[2]/div/section/div[3]/div/div/div/div/div[2]/div/div/div[1]/div/div/div/div["+str(x)+"]")
    link = driver.find_element(By.XPATH, "/html/body/div[1]/div/div[2]/div/main/div[1]/div[3]/div/div/div[1]/div[1]/div[2]/div/section/div[3]/div/div/div/div/div[2]/div/div/div[1]/div/div/div/div["+str(x)+"]/div/article/div[2]/div[5]/a/div/div[1]/img").get_attribute("href")
    driver.switch_to.window(driver.window_handles[1])

Error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[2]/div/main/div[1]/div[3]/div/div/div[1]/div[1]/div[2]/div/section/div[3]/div/div/div/div/div[2]/div/div/div[1]/div/div/div/div[1]/div/article/div[2]/div[5]/a/div/div[1]/img"}

Ajeet Verma
  • 2,938
  • 3
  • 13
  • 24
Sebastian Bardoff
  • 15
  • 4
  • 1
    You don't honestly expect any of us to trace through that ridiculous xpath to verify your findings, do you? Can you use a `class` or an `id` to come up with a search method that is not so incredibly delicate? Major web site makers change their HTML ALL THE TIME. – Tim Roberts Jun 23 '23 at 00:22
  • I don't really expect anything; I'm just wondering. I've spent two hours on this, and I still can't find anything wrong. I've verified that the HTML hasn't changed. I haven't found a way to use an ID or class to loop through and increment by one either. – Sebastian Bardoff Jun 23 '23 at 00:42
  • Can't you look for the `
    ` with the `data-a-target` of the clip you want, then fetch the first `
    ` from that?
     – Tim Roberts Jun 23 '23 at 01:43
  • I tried it just now, but that won't work either. It feels like it can't find any element at all when it comes to those clips. I'm not sure if it's due to how they are made in the HTML, but it just won't seem to work. Thanks for the tips though. – Sebastian Bardoff Jun 23 '23 at 01:55
  • How did you come up with that xpath? – John Gordon Jun 23 '23 at 03:12
  • I used a Chrome extension called Xpath Finder. When I click on an element, it provides me with the corresponding xpath.. I believe I clicked on the image to test it – Sebastian Bardoff Jun 23 '23 at 13:09

1 Answers1

0

Here's the neat and clean way of what You're trying to achieve, I suppose.

from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC

driver = Chrome()
driver.get('https://www.twitch.tv/xqc/videos?filter=clips&range=7d')

container = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[data-test-selector="content"]')))
videos = container.find_elements(By.CSS_SELECTOR, 'div[data-a-target^="clips-card-"]')
print(f"Total videos: {len(videos)}")

data = []
for video in videos:
    link_tag = video.find_element(By.CSS_SELECTOR, 'a[data-a-target="preview-card-image-link"]')

    link = link_tag.get_attribute('href')
    image = link_tag.find_element(By.TAG_NAME, 'img').get_attribute('src')

    data.append({'video_link': link, "video_image": image})

print(data)

output:

Total videos: 20
[{'video_link': 'https://www.twitch.tv/xqc/clip/BashfulCheerfulYogurtPJSalt-W8C1QC9Pvj_BjYgX?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/VF7m_1vAtUAvT74_wY2IFw/AT-cm%7CVF7m_1vAtUAvT74_wY2IFw-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/EvilMoldyWatercressRickroll-4B6d_LTIiD4x1gx1?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/dBY-0eAenfy0yIg-MdYMdQ/AT-cm%7CdBY-0eAenfy0yIg-MdYMdQ-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/IntelligentRelatedHumanTBTacoRight-ZXU6JQQKmTmMTTm0?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/nfTw3mV1OiUECpk5jE05tA/AT-cm%7CnfTw3mV1OiUECpk5jE05tA-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/ExquisiteSmilingAxeGrammarKing-D4Y2wd0rqp_vUs7G?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/xu_UFw-GNsHdK6qf71LOxg/AT-cm%7Cxu_UFw-GNsHdK6qf71LOxg-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/BelovedLitigiousFoxStoneLightning-4VUYUYIfIVQ8dCny?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/8ITTt-FVGFGCHyFUPHQp8g/AT-cm%7C8ITTt-FVGFGCHyFUPHQp8g-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/SpikyHeartlessLouseAsianGlow-xqkozwvTBQO74vX6?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/JWGfGMapKM8L0wUDu-L1-g/AT-cm%7CJWGfGMapKM8L0wUDu-L1-g-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/KitschyRelievedWoodpeckerDoubleRainbow-0btGlBb5cc-Ot3-7?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/E0ftKdG7RpQW-XHPjrd_uw/41521527833-offset-3214-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/HilariousCarefulLaptopGrammarKing-LjgdNV3i4_pO_XEs?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/zqBXIy44HTVd4uP6Tv1xLA/AT-cm%7CzqBXIy44HTVd4uP6Tv1xLA-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/StormyOpenJayTBTacoLeft-YFZJJutuwbsRMAXQ?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/mBx4QGBWSSbrbsG7ETAF0A/AT-cm%7CmBx4QGBWSSbrbsG7ETAF0A-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/TrappedPiercingCrabCmonBruh-kSJiYhKxcoKW-dnC?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/-SBpf5EiYmISsXKClvnZwg/AT-cm%7C-SBpf5EiYmISsXKClvnZwg-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/FaithfulBetterLocustBabyRage--gLgtcmgBu6aCy4m?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/EAXAffoC4LXJoAbumgsuJg/AT-cm%7CEAXAffoC4LXJoAbumgsuJg-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/HonorableExuberantDugongNinjaGrumpy-5asdA0LXl1HGuTdl?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/O5qjX-WrHQNC15dRxrk3ZQ/48691606397-offset-1430-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/KnottyHelpfulBulgogiLitty-GKSyTyi2JCynZLZt?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/_QyF3Biksk8otiQsp2S2bQ/AT-cm%7C_QyF3Biksk8otiQsp2S2bQ-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/EnthusiasticBetterClintPanicBasket-uQ3NtN2zWmVQ1Mec?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/2L562rkNzbTds1plUNTHnA/48691606397-offset-7252-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/SucculentSavoryWaffleDoubleRainbow-RRuJclEd6drGKHl6?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/GLvcfEUbiXJRHenaxih5eg/AT-cm%7CGLvcfEUbiXJRHenaxih5eg-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/TolerantWealthyReindeerFUNgineer-4SmvP6gr_ynuaj5U?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/G-tTXOEprz-oK_8p7mo-VA/AT-cm%7CG-tTXOEprz-oK_8p7mo-VA-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/FairFragileChickpeaTriHard-ABrG2iJ1QJSSzj4Q?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/vy-XRAAaxtoidUCUks4-EA/AT-cm%7Cvy-XRAAaxtoidUCUks4-EA-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/NeighborlyPolishedPterodactylLeeroyJenkins-y7ZcyabnwPbnpHwo?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/YcBDRsu2XMmm58D256R8KA/AT-cm%7CYcBDRsu2XMmm58D256R8KA-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/EsteemedTenaciousHyenaRalpherZ-NrjfJxexMST6_M2h?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/moOLbTa43ycXsTEk-vzNoQ/AT-cm%7CmoOLbTa43ycXsTEk-vzNoQ-preview-260x147.jpg'}, {'video_link': 'https://www.twitch.tv/xqc/clip/SpoopyUnusualRhinocerosThisIsSparta-1zV6nZGEkgjnqWmX?filter=clips&range=7d&sort=time', 'video_image': 'https://clips-media-assets2.twitch.tv/7RoHytc1ep2RhJII56lplw/AT-cm%7C7RoHytc1ep2RhJII56lplw-preview-260x147.jpg'}]

Steps followed:

  • Wait for the desired web element container holding all the data to get loaded to find/locate it.

    container = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[data-test-selector="content"]')))

  • Find all the individual target data points within the container. In the code above, we have used the strategy to locate elements using the attribute's partial value

    videos = container.find_elements(By.CSS_SELECTOR, 'div[data-a-target^="clips-card-"]')

  • Finally, iterate over the web element containing the list of data points to parse or extract the desired data.

I hope this solves the problem.

Ajeet Verma
  • 2,938
  • 3
  • 13
  • 24
  • It kinda worked, when I dont have the Selenium window selected I get this error: selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found. But when I dont have the window focused it works, its a really wierd. Thanks for the help – Sebastian Bardoff Jun 23 '23 at 13:06