4

Long story short : switched from Selenium to Requests(-html).

Works OK but not in every case.

Page : https://www.winamax.fr/paris-sportifs/sports/1/1/1

Upon load it charges dynamic content with english games (example : Sheffield United - West Ham).

But when I try to do this :

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.winamax.fr/paris-sportifs/1/1/1')
r.html.render()
print(r.html.text) # I also tried print(r.html.html)

the games don't show in the output.

Why ? Thanks !

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
jeremoquai
  • 101
  • 2
  • 10
  • Does this answer your question? [How to retrieve the values of dynamic html content using Python](https://stackoverflow.com/questions/17597424/how-to-retrieve-the-values-of-dynamic-html-content-using-python) – Ramon Medeiros Jan 09 '20 at 14:12
  • Simply because the "output" is not included in the page. In the Browser it is added to the DOM of the page by Javascript. Request does not run the Javascript. Selenium uses a browser to work. – Klaus D. Jan 09 '20 at 14:13
  • but requests-html wasn't it supposed to handle JavaScript Support ??? https://requests-html.kennethreitz.org/#javascript-support – jeremoquai Jan 09 '20 at 14:16
  • i tried to find the ajax json with the data, with no luck yet (unsure how to perform this) – jeremoquai Jan 09 '20 at 19:03
  • As OP says, it is the [Requests-HTML](https://requests-html.kennethreitz.org/) library he is using, which should be able to render JS and then inspect dynamically generated elements in the DOM, unlike Requests. Like OP, I've also experienced this to be quite sketchy though: On some sites, `render()` works and on others it doesn't. Would be great to know why. – amy Apr 02 '22 at 11:03

1 Answers1

4

add timeout, it should work, sorry this must be a comment but I cannot comment..

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.winamax.fr/paris-sportifs/sports/1/1/1')
r.html.render(timeout=20)
print(r.html.html)
session.close()
fardV
  • 198
  • 9
  • 1
    when im running, i get this `RuntimeWarning`: `C:\Applications__\python372\lib\site-packages\pyee\_base.py:81: RuntimeWarning: coroutine 'Browser._targetCreated' was never awaited f(*args, **kwargs) RuntimeWarning: Enable tracemalloc to get the object allocation traceback.` and then nothing is happening, it never stops from running. even with CTRL+C i cant stop the process. – alexzander Feb 26 '21 at 11:25
  • How can I use this with BeautifulSoup? – André Clérigo Sep 22 '21 at 18:20