0

I am trying to scrape course details from udemy, but it returns error as 403. I tried using user-agents but still it does'nt work.

import scrapy


class UdemySpider(scrapy.Spider):
    name = 'udemy'
    start_urls = [
        'https://www.udemy.com/api-2.0/search-courses/recommendation/?course_badge=beginners_choice&page_size=5&skip_price=true&q=python'
    ]

def parse(self, response):
    data = response.json()
    yield data

I have used scrapy-user-agent but still unable to scrape the data.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Mohd Abdul Azeem
  • 142
  • 2
  • 7
  • Have you tried looking at this? https://stackoverflow.com/questions/39202058/how-to-solve-403-error-in-scrapy/39202689, you basically just put headers into a `Request` object – A.K.M. Adib Apr 08 '21 at 03:58
  • Why does it return 403? Do you need to log in to see the courses? Have you tried that? – Jaanus Apr 08 '21 at 06:52
  • I can see the courses even without logging into it. I agree with the answer of @A.K.M.Adib and will look into it – Mohd Abdul Azeem Apr 08 '21 at 12:49

0 Answers0