Trying to get the html on an open page

Question

I am trying to make a bot that can play Cookie Clicker. I have successfully opened the website using the webbrowser module. When I use the developer tool to see the html I can see the information I want to obtain, such as how much money I have, how expensive items are ect. But when I try to get that information using the requests and beautifulsoup it instead gets the html of a new window. How can I make it so that I get the html of the already opened tab?

import webbrowser
webbrowser.open('https://orteil.dashnet.org/cookieclicker/')

from bs4 import BeautifulSoup
import requests

def scrape():
    html = requests.get('https://orteil.dashnet.org/cookieclicker/')
    print(html)

scrape()

I guess the webpages uses JavaScript to render the Page. So maybe try to open it in an webbrowser which renders the page first and then try to scrap. — Michael, Feb 02 '20 at 18:15
https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python&ved=2ahUKEwjp1ayUvLPnAhWryqYKHaUWC0AQjjgwAHoECAQQAQ&usg=AOvVaw02u5Epo0WelLQ4XWkzUCJC — Michael, Feb 02 '20 at 18:15

Joe · Accepted Answer · 2020-02-02T18:43:23.537

0

You can try to do this:

body_element = html.find_element_by_xpath("//body")    
body_content = body_element.get_attribute("innerHTML")    
print(body_content)

edited Feb 02 '20 at 18:43

answered Feb 02 '20 at 18:15

Joe

132
5

Trying to get the html on an open page

1 Answers1