0

usually I'm able to write a script that works for scraping, but I've been having some difficulty scraping this site for the table enlisted for this research project I'm working on. I'm planning to verify the script working on one State before entering the URL of my targeted states.

import requests
import bs4 as bs

url = ("http://programs.dsireusa.org/system/program/detail/284")
dsire_get = requests.get(url)
soup = bs.BeautifulSoup(dsire_get.text,'lxml')
table = soup.findAll('div', {'data-ng-controller': 'DetailsPageCtrl'})
print(table) 
#I'm printing "Table" just to ensure that the table information I'm looking for is within this sections

I'm not sure if the site is attempting to block people from scraping, but all the info that I'm looking to grab is within "&quot"if you look what Table outputs.

vlepore
  • 71
  • 7

2 Answers2

1

The text is rendered with JavaScript. First render the page with dryscrape

(If you don't want to use dryscrape see Web-scraping JavaScript page with Python )

Then the text can be extracted, after it has been rendered, from a different position on the page i.e the place it has been rendered to.

As an example this code will extract HTML from the summary.

import bs4 as bs
import dryscrape

url = ("http://programs.dsireusa.org/system/program/detail/284")
session = dryscrape.Session()
session.visit(url)
dsire_get = session.body()
soup = bs.BeautifulSoup(dsire_get,'html.parser')
table = soup.findAll('div', {'class': 'programSummary ng-binding'})
print(table[0])

Outputs:

<div class="programSummary ng-binding" data-ng-bind-html="program.summary"><p>
<strong>Eligibility and Availability</strong></p>
<p>
Net metering is available to all "qualifying facilities" (QFs), as defined by the federal <i>Public Utility Regulatory Policies Act of 1978</i> (PURPA), which pertains to renewable energy systems and combined heat and power systems up to 80 megawatts (MW) in capacity. There is no statewide cap on the aggregate capacity of net-metered systems.</p>
<p>
All utilities subject to Public ...
Dan-Dev
  • 8,957
  • 3
  • 38
  • 55
  • although this looks like it would work, dryscrape doesn't officially support windows so I am unable to use that. I'm going to follow the way laid out in that post you referenced without dryscape. – vlepore Jul 06 '17 at 17:40
  • That's why I included the link. Whether you use Dryscrape, Selenium, PyQt or anything else the methodology is the same. – Dan-Dev Jul 06 '17 at 18:05
1

So I finally managed to solve the issue, and successfuly grab the data from the Javascript page the code as follows worked for me if anyone encounters a same issue when trying to use python to scrape a javascript webpage using windows (dryscrape incompatible).

import bs4 as bs
from selenium import webdriver  
from selenium.common.exceptions import NoSuchElementException  
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome()
url = ("http://programs.dsireusa.org/system/program/detail/284")
browser.get(url)
html_source = browser.page_source
browser.quit()
soup = bs.BeautifulSoup(html_source, "html.parser")
table = soup.find('div', {'class': 'programOverview'})
data = []
for n in table.findAll("div", {"class": "ng-binding"}):
    trip = str(n.text)
    data.append(trip)
vlepore
  • 71
  • 7