Why is my BeautifulSoup code not working anymore when looking for something in Robinhood?

Question

I am working on a crypto bot and was almost done completing my project. For weeks, my team has searched for Robinhood API that would return the constantly updating price of BTC. The code below was working for the past week on the whole team's computer, however now it refuses to work. I have tried using different parsers, but cannot figure out what the problem is now. It was working fine for so long, and now suddenly refuses to work. Any help would be appreciated!

from bs4 import BeautifulSoup
import requests
import json


# returns value of bitcoin from https://robinhood.com/crypto/BTC using BeautifulSoup
def getPrice():
    price = ""

    response = requests.get("https://robinhood.com/crypto/BTC")  # Returns instance of Response class
    response.encoding = 'utf-8'  # Just in case the charset of response is not recognized

    # crypto: bs4.BeautifulSoup = BeautifulSoup(response.content, 'html.parser')
    # annotation format highlights what type of class the variable crypto is
    # https://stackoverflow.com/questions/51639332/use-of-colon-in-variable-declaration
    crypto = BeautifulSoup(response.content, "html.parser")

    for digit in crypto.find_all("span", {"class": "_9YsRP4ChsxbL9qzZnKv0K up"}):  # return type of find is object
        if digit.text != '$' and digit.text != ',':
            price += digit.text

    return float(price)

score 0 · Answer 1 · answered Jan 04 '20 at 03:41

If you look at the page source code. The contents you are looking for are loaded by js. Using requests.get will only get the page source, thus you won't get any content from the js engine. For that you have to use tools like webdriver from selenium module. You will have to install webdrivers too see here: https://selenium-python.readthedocs.io/installation.html.

Code:

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.firefox.options import Options

def getPrice():
    options = Options()
    options.add_argument('--headless')
    driver = webdriver.Firefox(options=options)
    driver.get('https://robinhood.com/crypto/BTC')
    crypto = BeautifulSoup(driver.page_source, 'html.parser')
    price=''
    for digit in crypto.find_all("span", {"class": "_9YsRP4ChsxbL9qzZnKv0K"}):
        if digit.text != '$' and digit.text != ',':
            price += digit.text
    return float(price)

Why is my BeautifulSoup code not working anymore when looking for something in Robinhood?

1 Answers1