0

I want to scrape Betway.com. However, when I try to request the page using the request library, I am getting a 403 Forbidden response.

I stumbled upon the following thread: Using headers with the Python requests library's get method It says that I can make my way around the problem using headers. I tried and it still didn't work. Here is my code:

import requests


url = "https://betway.com/fr/sports/cat/ufc---martial-arts"

headers = {'authority': "app.launchdarkly.com",
'method': "OPTIONS",
'path': "/sdk/goals/5ce3ca97d7d666081b7523a7",
'scheme': "https",
'accept': "*/*",
'accept-encoding': "gzip, deflate, br",
'accept-language': "fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7",
'access-control-request-headers': "x-launchdarkly-user-agent",
'access-control-request-method': "GET",
'origin': "https://betway.com",
'referer': "https://betway.com/",
'sec-fetch-dest': "empty",
'sec-fetch-mode': "cors",
'sec-fetch-site': "cross-site"}

result = requests.get(url, headers=headers)

Can you please help me find a solution. Also, it would be appreciated if you may share some easy resources on the field and what headers may mean for example. Thanks a lot.

lalaland
  • 379
  • 3
  • 15
  • You may need to add 'user-agent'. – Ouroborus Jul 19 '21 at 20:53
  • There's a lot more to it than just headers. Even after adding a user-agent as you are missing, the site could be protected by anti-scraping tech. I would remove just about all of those headers and try with just a user-agent. Most times all the additional stuff you included will be the thing getting you blocked. Start basic sending the least data as possible than move from there. This also will not work on sites with dynamic data loaded through JavaScript. – Voxum Jul 19 '21 at 21:00
  • @Ouroborus Thanks for the help. Using `user-agent ` worked. I have previously tried with a capital U and it didn't work. – lalaland Jul 19 '21 at 21:39
  • @Voxum You said that it is not going to work with dynamic data loaded through JavaScript. Is there a way to do it for that use case? – lalaland Jul 19 '21 at 21:40

0 Answers0