0

Is it possible to work around this "access denied" that I get when using this code on fantasy.premierleague.com? I tried using some random time delay but that doesn't seem to have any effect.

import requests 
import time
from bs4 import BeautifulSoup 
from selenium import webdriver

driver = webdriver.Firefox(executable_path=r'C:\Users\benja\Desktop\geckodriver.exe')

for y in range(7, 9):
    driver.get('https://fantasy.premierleague.com/leagues/181/standings/c?phase=1&page_new_entries=1&page_standings='+str(y))
    html = driver.execute_script("return document.documentElement.outerHTML")
    sel_soup = BeautifulSoup(html, 'html.parser')
    leaderboard = sel_soup.find('table', { 'class': 'Table-ziussd-1 hOInPp' })
    tbody = leaderboard.find('tbody')

    for tr in tbody.find_all('tr', {'class': 'StandingsRow-fwk48s-0 jRzimt'}):
        navn = tr.find_all('td')[1].find_all('a')[0].text.strip()
        link = tr.find_all('td')[1].find_all('a')[0]['href']
        print(navn, link)
    time.sleep(random.randint(5, 10))
Darendal
  • 843
  • 9
  • 29
Benjamin V. Jansen
  • 1
  • It's difficult to do much with so little information. It's possible that the site is detecting the fact that you're using an automated tool, and blocking you immediately, regardless of the number of requests. As an aside, is there a reason why you're mixing Selenium and BeautifulSoup like this? – AMC Feb 04 '20 at 19:55
  • Just some personal experience. I've never been able to get premierleague.com to work with ANY scraping strategy. I had to get my ref assignments from another site. Good luck. – blacktj Feb 04 '20 at 21:06
  • Without automation, you also cannot access the url by manual open browser. – Yun Feb 05 '20 at 03:05
  • @AMC Hard to say, but before I added the loop I didn't get any errors, so I would think there is some detection of multiple requests. As to why I'm using Selenium and BeautifulSoup like this: I don't know, I just tried to make it work. – Benjamin V. Jansen Feb 05 '20 at 10:02
  • @blacktj Yeah, it seems like a lot of people are having troubles with this. – Benjamin V. Jansen Feb 05 '20 at 10:05
  • Luckily, I found a solution to my particular case. Using API: https://medium.com/@YourMumSaysWhat/how-to-get-data-from-the-fantasy-premier-league-api-4477d6a334c3 – Benjamin V. Jansen Feb 05 '20 at 10:06

0 Answers0