-1

i am trying to get the code of a website using Python. The problem is that when i try to create a GET request using cloudscraper, it returns the instant code generated in HTML.

On this website, some code appears after the page has been rendered. How can i make it so that the scrapper return the code after a few seconds after the page has been opened?

This is my code:

  scraper = cloudscraper.create_scraper()  
        content = scraper.get("link").text

I have already tried to add a delay in the create_scrapper but it does not seem to work. Thank you for the help!

Andrei Marin
  • 616
  • 1
  • 6
  • 16
  • The content might be loaded by another request, you could either find this request an reproduce it instead. Or JS modify the DOM after the http response, in this case, Selenium might be more suited – Ant0ine64 Aug 28 '23 at 13:01
  • Does this answer your question? [Can scrapy be used to scrape dynamic content from websites that are using AJAX?](https://stackoverflow.com/questions/8550114/can-scrapy-be-used-to-scrape-dynamic-content-from-websites-that-are-using-ajax) – He3lixxx Aug 29 '23 at 12:31

1 Answers1

0

I am not familiar with cloudscraper, it seems to be specifically to bypass Cloudflares anti-bot page, did you choose this package for that reason?

If not, did you checkout these threads? Both propose Selenium, which is a browser automation tool.

Wait for page to load before scraping

Wait page to load before getting data with requests.get in python 3

kahobe
  • 36
  • 4