Scraping top 250 movies from imdb website

Question

I want to scrape top 250 movies from imdb website by beautiful soup in python but it returns nothing in my output.

import requests
from bs4 import BeautifulSoup


url = "https://www.imdb.com/chart/top/?ref_=nv_mv_250"
response = requests.get(url)
html_content = response.content
soup = BeautifulSoup(html_content, "html.parser")
movies = soup.find_all(
    "li", class_="ipc-metadata-list-summary-item sc-bca49391-0 eypSaE cli-parent")
for x in movies:
    print(x.h3.text)

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. — Community, Jun 29 '23 at 21:10

score 2 · Accepted Answer · answered Jun 29 '23 at 21:37

Your problem is related with Python requests. 403 Forbidden

So, I've tried the same as suggested in the StackOverflow thread and work. This is the code that works for me:

import requests
from bs4 import BeautifulSoup


headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
url = "https://www.imdb.com/chart/top/?ref_=nv_mv_250"
response = requests.get(url, headers=headers)
html_content = response.content
soup = BeautifulSoup(html_content, "html.parser")
movies = soup.find_all(
    "li", class_="ipc-metadata-list-summary-item sc-bca49391-0 eypSaE cli-parent")
for x in movies:
    print(x.h3.text)

Scraping top 250 movies from imdb website

1 Answers1