How to click a link within youtube comment using an user agent through Selenium and Python

Question

I am writing a script that clicks a link in the youtube comment, it works fine, but when I combine it with the user agent it doesn't work, can someone help me?

example : link

Html :

<a class="yt-simple-endpoint style-scope yt-formatted-string" spellcheck="false" href="/redirect?redir_token=ix1uBK3TO3bxXdcT7EvDFp-vI9p8MTU0NjQxNTExOEAxNTQ2MzI4NzE4&amp;event=comments&amp;q=https%3A%2F%2Fjulissars.itworks.com%2F&amp;stzid=Ugw6ip_QkzwyJPIq3bp4AaABAg" rel="nofollow">https://julissars.itworks.com&#65279;</a>

Code trial ( without user agent , it works ) :

from selenium import webdriver
from fake_useragent import UserAgent
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("https://www.youtube.com/watch?v=UJezMYvf8Ss&lc=Ugw6ip_QkzwyJPIq3bp4AaABAg")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='yt-simple-endpoint style-scope yt-formatted-string' and contains(., 'julissars')]"))).click()

Code trial ( with user agent , it doesn't work ) :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent

useragent = UserAgent()
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", useragent.random)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("https://www.youtube.com/watch?v=UJezMYvf8Ss&lc=Ugw6ip_QkzwyJPIq3bp4AaABAg")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='yt-simple-endpoint style-scope yt-formatted-string' and contains(., 'julissars')]"))).click()

Answer 1

To click on the desired comment with text as https://julissars.itworks.com within the url you need to induce WebDriverwait for the element to be clickable and you can use the following solution using useragent through Selenium and Python:

• Code Block:

  from selenium import webdriver
  from fake_useragent import UserAgent
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  ua = UserAgent()
  options = webdriver.ChromeOptions()
  userAgent = ua.random
  print(userAgent)
  options.add_argument('user-agent=' + userAgent)
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://www.youtube.com/watch?v=UJezMYvf8Ss&lc=Ugw6ip_QkzwyJPIq3bp4AaABAg")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='yt-uix-sessionlink  ' and contains(@href, 'julissars')]"))).click()
  

• Console Output:

  Mozilla/5.0 (X11; NetBSD) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36
  

• You can find a detailed discussion in Way to change Google Chrome user agent in Selenium?

Answer 2

Google very actively blocks bots from interacting with most of their properties. If the user agent string you're sending really is random (that is, it doesn't correspond to a real UA that a browser might send), you're probably getting caught in their bot detection algorithms.

Thus, your click is probably "working" in the sense that it's actually triggering an event in the web browser... but Google is probably ignoring the request.

Answer 3

The linked webpage doesn’t support all browsers (user agents). If it’s your problem you can choose a few supported user agents and rotate between them (for example with random.choice()). You need to update your xpath as well.

from selenium import webdriver
from fake_useragent import UserAgent
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

useragent = UserAgent()
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", useragent.random)
driver = webdriver.Firefox(firefox_profile=profile)

driver.get("https://www.youtube.com/watch?v=UJezMYvf8Ss&lc=Ugw6ip_QkzwyJPIq3bp4AaABAg")

xpath = '/html/body/div[2]/div[4]/div/div[5]/div[2]/div[2]/div/div[2]/div[5]/div/div[2]/section[1]/div[1]/div[2]/div[2]/div[1]/a'

try:
    element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, xpath)))
finally:
    element.click()