Python Selenium find_elements_by_class_name Error

Question

I am scraping a google page that has returned links to Linkedin profiles.

I want to collect the links on a page and put them in a python list.

Problem is I can't seem to properly extract them from the page and I don't know why.

Google source code looks like this:

The page displays 10 of the following:

Mary Smith - Director of Talent Acquisition ...
https://www.linkedin.com › marysmith
Anytown, Arizona 500+ connections ... Experienced Talent Acquisition Director, with a 
demonstrated history of working in the marketing and advertising ...

The source code looks like this:

<div data-hveid="CAIQAA" data-ved="2ahUKEwjLv6HMr4HmAhWluVkKHfjfA1EQFSgAMAF6BAgCEAA">
   <div class="rc"> 
       <div class="r">
           <a href="https://www.linkedin.com/in/marysmith" ping="/url?sa=t&amp;source=web&amp;rct=j&amp;url=https://www.linkedin.com/in/marysmith&amp;ved=2ahUKEwjLv6HMr4HmAhWluVkKHfjfA1EQFjABegQIAhAB">
               <h3 class="LC20lb"><span class="S3Uucc">Mary Smith - Director of Talent Acquisition, Culture Curator ...</span></h3><br>
               <div class="TbwUpd">
                   <cite class="iUh30 bc">https://www.linkedin.com › marysmith</cite>
              </div>
           </a>
           ...

In my script I'm using Selenium and find_element_by_class_name()to collect all the instances of the links to Linkedin. The one in the above example is https://www.linkedin.com › marysmith. It is one line of code where I use driver.find_element_by_class_name() with the particular class name:

linkedin_urls = driver.find_element_by_class_name("iUh30 bc")

However I get the following error:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[name="iUh30 bc"]"}

I've tried various permutations and other classes but it won't work. If I use the X_Path for one of those links the script WILL return that single link.

What am I doing wrong?

Have you tried "iUh30" or "bc" individually? – Alec McGail Nov 24 '19 at 00:47 — Alec McGail, Nov 24 '19 at 00:47
Tried both separately. Neither worked. – Windstorm1981 Nov 24 '19 at 00:56 — Windstorm1981, Nov 24 '19 at 00:56
I also tried "iUh30.bc" and it didnt work – Windstorm1981 Nov 24 '19 at 00:59 — Windstorm1981, Nov 24 '19 at 00:59

score 1 · Answer 1 · edited Apr 07 '20 at 12:06

1

Websites like Google and Facebook use an AI to construct the pages sources and assign random classes that's why you are getting no such element because every time you load that page the class's value varies To solve this issue try to use constant tags or attributes.

Try something like:

#<cite class="iUh30 bc">https://www.linkedin.com › mary-smith-mckenzie-8b660799</cite>
driver.find_elements_by_xpath("//cite[contains(text(),'›') and contains(text(),'linkedin.com')]")

edited Apr 07 '20 at 12:06

Martijn Pieters

1,048,767
296
4,058
3,343

answered Nov 24 '19 at 03:46

Ahmed Soliman

1,662
1
11
16

I get the following error when I try your solution `SyntaxError: Non-ASCII character '\xe2' in file C:/Users//Documents/Python Scripts/jobscrape/jobscrape.py on line 39, but no encoding declared;` Thoughts? – Windstorm1981 Nov 24 '19 at 15:08
@Windstorm1981 have a look at this [Python “SyntaxError: Non-ASCII character '\xe2' in file”](https://stackoverflow.com/questions/21639275/python-syntaxerror-non-ascii-character-xe2-in-file) – Ahmed Soliman Nov 25 '19 at 01:26

score 0 · Answer 2 · answered Nov 24 '19 at 01:45

0

That method is known to be buggy. Try:

driver.find_element_by_css_selector(".iUh30.bc")

answered Nov 24 '19 at 01:45

pguardiario

53,827
19
119
159

linkedin_urls still comes up empty and non-iterable. Thoughts? – Windstorm1981 Nov 24 '19 at 15:09
find_element methods return 1 element, find_elements methods return a list – pguardiario Nov 25 '19 at 00:55

Python Selenium find_elements_by_class_name Error

2 Answers2