Get player prices in fantasy football using Selenium

Question

I am looking to scrape player prices on https://www.fanteam.com/participate/138905/new/e30= using Python and Selenium libraries. I have used the following code:

url = 'https://www.fanteam.com/participate/138905/new/e30='
options = webdriver.ChromeOptions()
options.add_argument('--lang=en')

driver = webdriver.Chrome(chrome_options=options)
driver.get(url)

But I can't get all the players with prices, because I can't find any element on the page(see the picture below players with prices). There is HTML of this site:

<!DOCTYPE html>
<html lang="en">

<head>
  <script type='text/javascript'>
  </script>
  <meta charset="UTF-8">
  <link rel="shortcut icon" type="image/x-icon" href="/assets/favicon.ico">
  <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no, minimal-ui">
  <meta name="mobile-web-app-capable" content="yes">
  <meta property="og:title" content="FanTeam: The home of Fantasy Sports">
  <meta property="og:description" content="Create Your Daily Fantasy Team, Play &amp; Win Cash!">
  <meta property="og:site_name" content="FanTeam">
  <meta property="og:image:width" content="300">
  <meta property="og:image:height" content="300">
  <meta property="og:url" content="https://www.fanteam.com/participate/138905/new/e30=">
  <meta property="og:image" content="https://www.fanteam.com/assets/og-banner.png">
  <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,300,600,700,800&amp;subset=latin,cyrillic,cyrillic-ext,latin-ext" rel="stylesheet" type="text/css">
  <link rel="manifest" href="/manifest.json">
  <script>
    (function(getDescriptor) {
      Object.getOwnPropertyDescriptor = function(obj, key) {
        var descriptor = getDescriptor.apply(this, arguments)
        if (!descriptor && obj === window && key == "showModalDialog") {
          return {}
        }
        return descriptor
      }
    }(Object.getOwnPropertyDescriptor));
  </script>
  <style>
  </style>
  <title>FanTeam - Daily Fantasy & Betting</title>
</head>

<body>
  <ft-cookie-warning></ft-cookie-warning>
  <main>
    <ft-header logo="fanteam-logo.svg" logosmall="logosmall.svg"></ft-header>
    <section class="ft-view-port-wrapper">
      <view-port></view-port>
    </section>
    <ft-footer tabindex="-1" logo="fanteam-logo.svg"></ft-footer>
    <ft-push-receiver></ft-push-receiver>
    <ft-olark></ft-olark>
  </main>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/webcomponentsjs/1.0.6/webcomponents-lite.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/babel-polyfill/6.26.0/polyfill.min.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/fetch/2.0.3/fetch.min.js"></script>
  <script src="/build/application-b8ab977b2a.js" data-root="https://fanteam-game.api.scoutgg.net" data-ws="https://fanteam-game.ws.scoutgg.net" data-auth-url="" data-white-label="fanteam" data-olark="8903-397-10-7512" data-google-analytics="UA-55860585-1"
    data-asset-host="https://d34h6ikdffho99.cloudfront.net" data-vapid-public-key="BH8zySo8DKTd9EY0koPSAmA7fo58QTVuFjcB4hTp95WDu21l4dwjckigl0hpYBgeS-6h2kbMtfbXw4u4097wK3w" data-scoutcc="https://scoutcc.scoutgg.net" data-payment-url="https://globpay.fantasy.solutions/v1"
    data-projection-url="https://betflex-projection.api.scoutgg.net//api/v1" data-sportsbook-path="https://stage.fenixplayground.es/apuestas/mobilegoto.aspx" data-service-worker="sw.js"></script>
</body>

</html>

Any code like

el = driver.find_element_by_xpath("//div[@class='player-list']")

return me the error:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[@class='player-list']"}

But when I inspect an element I can see it in the browser. How to click any element on the page?

Please, post the HTML of the page instead of the screenshot. — Mate Mrše, Mar 19 '19 at 09:15
Please don't post the entire HTML of the page. Post only the relevant portion of the HTML. Also, the code you have posted doesn't have anything past the loading of the URL. You need to add a [mcve] so we can see what you are doing. — JeffC, Mar 19 '19 at 13:57

Sameer Arora · Accepted Answer · 2019-03-19T11:16:07.787

2

The website you are trying to scrape has a shadow-DOM in its html and any html present inside it cannot be accessed and that is the reason you are getting NoSuchElementException.
Currently, selenium does not support the shadow DOM automation, so you need to use javascript in this case to scrape the data.

To get the data using javascript, you can use:

JavascriptExecutor js = (JavascriptExecutor) driver;  
String return_value = (String) js.execute_script("return document.getElementByXpath('xpath').innerHTML");

References for the shadow DOM:
https://medium.com/rate-engineering/a-guide-to-working-with-shadow-dom-using-selenium-b124992559f
https://www.seleniumeasy.com/selenium-tutorials/accessing-shadow-dom-elements-with-webdriver

edited Mar 19 '19 at 11:16

answered Mar 19 '19 at 09:38

Sameer Arora

4,439
3
10
20

so you mean I should use only javascript, not Python? – Art Mich Mar 19 '19 at 11:12
Not javascript language, you need to use JavaScriptExecutor using python, it directly operates on the html of the page and by that you can get the data from it. You can refer these links on how to use javascript executor: https://stackoverflow.com/questions/7794087/running-javascript-in-selenium-using-python and https://www.guru99.com/execute-javascript-selenium-webdriver.html – Sameer Arora Mar 19 '19 at 11:15
Not yet( I use only Python, so I haven't applied your solution yet. – Art Mich Mar 19 '19 at 11:30
I am not asking you to use any other language, You can use JavaScriptExecutor with python only. JavaScriptExecutor is a way to select and get details from the html structure directly. Though you can wait for other answers if you want to but i don't think there would be any other solution to this. – Sameer Arora Mar 19 '19 at 11:32
@ArtMich did you find any other way to do it ? – Sameer Arora Mar 20 '19 at 09:50

Get player prices in fantasy football using Selenium

1 Answers1