1

I'm trying to scrape a part of page that is generated by JS so the value isn't present in the source code.

I am using Selenium (first time) with BeautifulSoap (familiar with this).

CAUTION: The URL is an adult site!

Here is my code:

import sys
import urllib2
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup
import re
import pandas as pd
import os

url = "https://www.kink.com/shoot/5540"

options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.implicitly_wait(30)
driver.get(url)

soup = BeautifulSoup(driver.page_source, 'lxml')

print url

print soup

However the soup I am getting doesn't contain the JS generated bit.

I specifically need the rating part only that says: "Average Rating: 58 ratings".

What am I missing?

michaelgam85
  • 11
  • 2
  • Related: https://stackoverflow.com/questions/28289699/python-web-scraping-for-javascript-generated-content – wcarhart Apr 15 '19 at 10:07

0 Answers0