0

In this html

<div class="css-rcl8e5">
 <span class="css-wn0avc">
   Salary
   :
 </span>
  <span class="css-47jx3m">
    <span class="css-4xky9y">***2000 to 3000 EGP Per Month***</span>
  </span>
</div>

I am trying to get the salary value using find()

for link in links:
    result = requests.get(link)
    src = result.content
    soup = BeautifulSoup(src, "lxml")
    salaries = soup.find("span", {"class":"css-4xky9y"})
    print(salaries)
    salary.append(salaries)

It returns none. I tried find_all() but it returns an empty list. However, if I change the class to another one it works just fine. Any idea why that is and what can I do to solve this issue other than using selenium?

EDS
  • 2,155
  • 1
  • 6
  • 21
eelbayar
  • 1
  • 2
  • I tried your code with your html sample and it works... so what about your requests? try to pass a user-agent parameter – cards Sep 04 '21 at 21:43
  • headers = {'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/87.0'} response = requests.get(url, headers=headers) – cards Sep 04 '21 at 21:52
  • The real HTML must be different than what you posted. – John Gordon Sep 04 '21 at 21:52
  • other than selenium i saw requests_html, haven't tried yet but looks very cool – diggusbickus Sep 04 '21 at 21:58
  • Does the website use javascript to dynamically alter the page content? `requests` doesn't do javascript. – John Gordon Sep 04 '21 at 22:33
  • Could you post what is the URL ? – Ram Sep 05 '21 at 06:34
  • Does this answer your question? [can't get all span tag inside div element beautifulsoup](https://stackoverflow.com/questions/68999524/cant-get-all-span-tag-inside-div-element-beautifulsoup) - Think you should give selenium a chance :) – HedgeHog Sep 05 '21 at 09:11
  • share me `website link` then I will help you – Arslan Aziz Sep 20 '21 at 13:05

0 Answers0