Scrapping using Selenium and Python

Question

I am trying to scrape some data off a tweet

from selenium import webdriver
!pip install kora -q
from kora.selenium import wd

link = 'https://twitter.com/psalmcrypt/status/1442460419612835840?s=19'

wd.get(link)


for a in wd.find_element(By.XPATH , '//*[@id="id__armfsi35bm"]/div[2]/div/div/a'):
  promoter = a.get_attribute('href')

print(promoter)

I am getting the following error-

I am copying the xpath from the inspector window

Which information are you attempting to scrape exactly? – undetected Selenium Aug 05 '22 at 12:49 — undetected Selenium, Aug 05 '22 at 12:49

undetected Selenium · Answer 1 · 2022-08-05T14:09:54.967

0

The for loop is used to iterate over a sequence i.e. either a list, a tuple, a dictionary, a set or a string.

In your usecase presumably it should have been a list of WebElements. To return a list instead of find_element() you'd need find_elements() as follows:

for a in wd.find_elements(By.XPATH , 'xpath_of_the_elements'):

edited Aug 05 '22 at 14:09

answered Aug 05 '22 at 12:56

undetected Selenium

183,867
41
278
352

i use this in other way. driver.find_elements(By.XPATH , 'xpath/of_the/elements/*') This return a list of web elements so you can work by his positions. is pretty useful when you have troubles getting an element – Carlost Aug 05 '22 at 16:27

Scrapping using Selenium and Python

1 Answers1