0

I am using the requests library to retrieve data from nitrotype.com/racer/insert_name_here about a user's progress using the following code:

import requests

base_url = 'https://www.nitrotype.com/racer/'
name = 'test'
url = base_url + name

page = requests.get(url)
print(page.text)

However my problem is that this retrieves data from the loading screen, I want the data after the loading screen. Is it possible to do this and how?

Systematic Error
  • 18
  • 4

1 Answers1

0

This is likely because of dynamic loading and can easily be navigated by using selenium or pyppeteer.

In my example, I have used pyppeteer to spawn a browser and load the javascript so that I can attain the required information.

Example:

import pyppeteer
import asyncio

async def main():
    # launches a chromium browser, can use chrome instead of chromium as well.
    browser = await pyppeteer.launch(headless=False)
    # creates a blank page
    page = await browser.newPage()
    # follows to the requested page and runs the dynamic code on the site.
    await page.goto('https://www.nitrotype.com/racer/tupac')
    # provides the html content of the page
    cont = await page.content()
    return cont

# prints the html code of the user profiel: tupac
print(asyncio.get_event_loop().run_until_complete(main()))
Souldiv
  • 145
  • 1
  • 5
  • I got this error: Traceback (most recent call last): File "main.py", line 16, in print(asyncio.get_event_loop().run_until_complete()) TypeError: run_until_complete() missing 1 required positional argument: 'future' – Systematic Error Jun 09 '20 at 14:47