-1
from bs4 import BeautifulSoup
import requests


html_text=requests.get('https://www.upwork.com/nx/jobs/search/?q=Lead%20Generation&sort=recency').text
soup=BeautifulSoup(html_text,'lxml')
jobs=soup.find_all('div', class_="job-description-line-clamp")

print(jobs)

I only get []. Why?

I want to scrape upwork jobs.

Jeppe
  • 1,830
  • 3
  • 24
  • 33
İqbal Mirzeyev
  • 1
  • Looks like the data on that page is dynamically fetched and rendered, and hence, the javascript within the HTML needs to execute for the data to exist. The HTML itself does not contain the data (yet). You should look into scraping using e.g. a browser. There's a lot of threads on StackOverflow using `Selenium` to achieve that, e.g. this: https://stackoverflow.com/questions/8650999/using-python-with-selenium-to-scrape-dynamic-web-pages – Jeppe Apr 29 '23 at 07:28
  • In addition, you have to read the Terms of Use of a site you're about to scrape. Upwork requires written permission to scrape their sites - see item 3.5 in their Terms of Use: https://www.upwork.com/legal#terms-of-use – Jeppe Apr 29 '23 at 07:32

2 Answers2

0

When you try to send a request, the site says Enable JavaScript and cookies to continue.

Which says you need to enable JavaScript. The requests library does not support this, so I advise you to use a browser emulator. For example selenium.

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.upwork.com/nx/jobs/search/?q=Lead%20Generation&sort=recency")

bs = BeautifulSoup(driver.page_source, "lxml")

jobs = bs.find_all('div', {"data-test": "job-description-line-clamp"})

print(jobs)

Note: I changed find tag to data-test

Wilidon
  • 1
  • 1
-1

Try using request headers. There is a GraphQl request that fetches jobs. I tried fetching "Best matches" job on Upwork.

. The screenshot is attached for reference.

Ajeet Verma
  • 2,938
  • 3
  • 13
  • 24
Usman Fawad
  • 1
  • 2
  • Your code is badly visible. The best practice is copy code as text and apply Stackoverflow formating. – Sergey Zaykov May 01 '23 at 17:15
  • Hello @SergeyZaykov, I did not share the code but just pasted a screenshot of a chrome inspector. – Usman Fawad May 02 '23 at 14:45
  • Please [don’t post images of code, error messages, or other textual data.](https://meta.stackoverflow.com/questions/285551/why-not-upload-images-of-code-on-so-when-asking-a-question/285557#285557) – tripleee May 03 '23 at 11:03