2

I'm trying to pull a specific table from this Wunderground page: https://www.wunderground.com/history/daily/us/ma/nantucket/KACK/date/2018-7-29

In plain english, the table is called "Daily Observations".

From inspecting the page, it looks like the table id is history-observation-table

I've tried using BeautifulSoup, but every way I can think of to find the table (or ANY tables) does not work.

page = requests.get('https://www.wunderground.com/history/daily/us/ma/nantucket/KACK/date/2018-7-29').text


soup = bs(page.content,'html.parser')

soup.find_all("table")

The result is nothing/empty. I can find the title, and the divs, but not if I look for specific class divs. Why can't I pull this table?

bearcub
  • 313
  • 4
  • 11
  • 2
    If the website uses on-page javascript to generate content, `requests` will never see it. In that case you'll need something like selenium. – John Gordon May 13 '19 at 21:33

3 Answers3

3

The page is rendering the table with javascript, so BeautifulSoup will not know it is there. You can use selenium to get the correct page source and feed that into a soup object though!

You will need to install selenium at which point your script would become:

from bs4 import BeautifulSoup as bs
from selenium import webdriver
import time

browser = webdriver.Chrome() # or some other browser
browser.get('https://www.wunderground.com/history/daily/us/ma/nantucket/KACK/date/2018-7-29')
time.sleep(2)
soup = bs(browser.page_source, 'html.parser')

print(soup.find_all("table"))

It would also be better to replace time.sleep() with selenium waits

When I run the above script it outputs a lengthy:

[<table _ngcontent-c14="" id="stationselector_table">
<tbody _ngcontent-c14="">
<!-- -->
</tbody>
</table>, <table _ngcontent-c7="">
<!-- --><!-- -->
<thead _ngcontent-c7="">
<tr _ngcontent-c7="">
<th _ngcontent-c7="">Temperature (° F)</th>
<!-- --><td _ngcontent-c7="">Actual</td><td _ngcontent-c7="">Historic Avg.</td><td _ngcontent-c7="">Record</td>
<td _ngcontent-c7="" style="height: 5px; width:10px;">
<svg _ngcontent-c7="" height="5" style="display: block" width="10" xmlns="http://www.w3.org/2000/svg"><polygon _ngcontent-c7="" fill="#000000" points="0,5 5,0 10,5"></polygon></svg>
</td>
</tr>
</thead>
<tbody _ngcontent-c7="">
<!-- --><tr _ngcontent-c7="">
<th _ngcontent-c7="">High Temp</th>
<!-- --><td _ngcontent-c7="">80</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">90</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Low Temp</th>
<!-- --><td _ngcontent-c7="">66</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">53</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Day Average Temp</th>
<!-- --><td _ngcontent-c7="">74</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr>
</tbody>
<!-- -->
<thead _ngcontent-c7="">
<tr _ngcontent-c7="">
<th _ngcontent-c7="">Precipitation (Inches)</th>
<!-- --><td _ngcontent-c7="">Actual</td><td _ngcontent-c7="">Historic Avg.</td><td _ngcontent-c7="">Record</td>
<td _ngcontent-c7="" style="height: 5px; width:10px;">
<svg _ngcontent-c7="" height="5" style="display: block" width="10" xmlns="http://www.w3.org/2000/svg"><polygon _ngcontent-c7="" fill="#000000" points="0,5 5,0 10,5"></polygon></svg>
</td>
</tr>
</thead>
<tbody _ngcontent-c7="">
<!-- --><tr _ngcontent-c7="">
<th _ngcontent-c7="">Precipitation</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">2.4</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Month to Date</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Year to Date</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr>
</tbody>
<!-- -->
<thead _ngcontent-c7="">
<tr _ngcontent-c7="">
<th _ngcontent-c7="">Degree Days (° F)</th>
<!-- --><td _ngcontent-c7="">Actual</td><td _ngcontent-c7="">Historic Avg.</td><td _ngcontent-c7="">Record</td>
<td _ngcontent-c7="" style="height: 5px; width:10px;">
<svg _ngcontent-c7="" height="5" style="display: block" width="10" xmlns="http://www.w3.org/2000/svg"><polygon _ngcontent-c7="" fill="#000000" points="0,5 5,0 10,5"></polygon></svg>
</td>
</tr>
</thead>
<tbody _ngcontent-c7="">
<!-- --><tr _ngcontent-c7="">
<th _ngcontent-c7="">Heating Degree Days</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">HDD Month to Date</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">HDD Since July 1</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Cooling Degree Days</th>
<!-- --><td _ngcontent-c7="">9</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">CDD Month to Date</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">CDD Year to Date</th>
<!-- --><td _ngcontent-c7="">0</td><td _ngcontent-c7="">0</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Growing Degree Days</th>
<!-- --><td _ngcontent-c7="">24</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr>
</tbody>
<!-- -->
<thead _ngcontent-c7="">
<tr _ngcontent-c7="">
<th _ngcontent-c7="">Dew Point (° F)</th>
<!-- --><td _ngcontent-c7="">Actual</td><td _ngcontent-c7="">Historic Avg.</td><td _ngcontent-c7="">Record</td>
<td _ngcontent-c7="" style="height: 5px; width:10px;">
<svg _ngcontent-c7="" height="5" style="display: block" width="10" xmlns="http://www.w3.org/2000/svg"><polygon _ngcontent-c7="" fill="#000000" points="0,5 5,0 10,5"></polygon></svg>
</td>
</tr>
</thead>
<tbody _ngcontent-c7="">
<!-- --><tr _ngcontent-c7="">
<th _ngcontent-c7="">Dew Point</th>
<!-- --><td _ngcontent-c7="">70</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">High</th>
<!-- --><td _ngcontent-c7="">72</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Low</th>
<!-- --><td _ngcontent-c7="">65</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Average</th>
<!-- --><td _ngcontent-c7="">70</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr>
</tbody>
<!-- -->
<thead _ngcontent-c7="">
<tr _ngcontent-c7="">
<th _ngcontent-c7="">Wind (MPH)</th>
<!-- --><td _ngcontent-c7="">Actual</td><td _ngcontent-c7="">Historic Avg.</td><td _ngcontent-c7="">Record</td>
<td _ngcontent-c7="" style="height: 5px; width:10px;">
<svg _ngcontent-c7="" height="5" style="display: block" width="10" xmlns="http://www.w3.org/2000/svg"><polygon _ngcontent-c7="" fill="#000000" points="0,5 5,0 10,5"></polygon></svg>
</td>
</tr>
</thead>
<tbody _ngcontent-c7="">
<!-- --><tr _ngcontent-c7="">
<th _ngcontent-c7="">Max Wind Speed</th>
<!-- --><td _ngcontent-c7="">9</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Visibility</th>
<!-- --><td _ngcontent-c7="">10</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr>
</tbody>
<!-- -->
<thead _ngcontent-c7="">
<tr _ngcontent-c7="">
<th _ngcontent-c7="">Sea Level Pressure (Hg)</th>
<!-- --><td _ngcontent-c7="">Actual</td><td _ngcontent-c7="">Historic Avg.</td><td _ngcontent-c7="">Record</td>
<td _ngcontent-c7="" style="height: 5px; width:10px;">
<svg _ngcontent-c7="" height="5" style="display: block" width="10" xmlns="http://www.w3.org/2000/svg"><polygon _ngcontent-c7="" fill="#000000" points="0,5 5,0 10,5"></polygon></svg>
</td>
</tr>
</thead>
<tbody _ngcontent-c7="">
<!-- --><tr _ngcontent-c7="">
<th _ngcontent-c7="">Sea Level Pressure</th>
<!-- --><td _ngcontent-c7="">30.11</td><td _ngcontent-c7="">-</td><td _ngcontent-c7="">-</td>
</tr>
</tbody>
<!-- -->
<thead _ngcontent-c7="">
<tr _ngcontent-c7="">
<th _ngcontent-c7="">Astronomy</th>
<!-- --><td _ngcontent-c7="">Day Length</td><td _ngcontent-c7="">Rise</td><td _ngcontent-c7="">Set</td>
<td _ngcontent-c7="" style="height: 5px; width:10px;">
<svg _ngcontent-c7="" height="5" style="display: block" width="10" xmlns="http://www.w3.org/2000/svg"><polygon _ngcontent-c7="" fill="#000000" points="0,5 5,0 10,5"></polygon></svg>
</td>
</tr>
</thead>
<tbody _ngcontent-c7="">
<!-- --><tr _ngcontent-c7="">
<th _ngcontent-c7="">Actual Time</th>
<!-- --><td _ngcontent-c7="">14h 28m</td><td _ngcontent-c7="">5:33 AM</td><td _ngcontent-c7="">8:02 PM</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Civil Twilight</th>
<!-- --><td _ngcontent-c7=""></td><td _ngcontent-c7="">5:02 AM</td><td _ngcontent-c7="">8:33 PM</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Nautical Twilight</th>
<!-- --><td _ngcontent-c7=""></td><td _ngcontent-c7="">4:23 AM</td><td _ngcontent-c7="">9:12 PM</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Astronomical Twilight</th>
<!-- --><td _ngcontent-c7=""></td><td _ngcontent-c7="">3:39 AM</td><td _ngcontent-c7="">9:56 PM</td>
</tr><tr _ngcontent-c7="">
<th _ngcontent-c7="">Moon: waning gibbous</th>
<!-- --><td _ngcontent-c7=""></td><td _ngcontent-c7="">9:13 PM</td><td _ngcontent-c7="">7:03 AM</td>
</tr>
</tbody>
</table>, <table _ngcontent-c17="" class="tablesaw-sortable" id="history-observation-table">
<thead _ngcontent-c17="">
<tr _ngcontent-c17="">
<!-- --><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-cell-persist tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Time</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-cell-persist tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Temperature</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Dew Point</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Humidity</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Wind</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Wind Speed</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Wind Gust</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Pressure</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Precip.</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Precip Accum</button>
</span></ngsaw-header>
</th><th _ngcontent-c17="">
<ngsaw-header _ngcontent-c17=""><span class="tablesaw-sortable-head">
<button class="tablesaw-sortable-btn">Condition</button>
</span></ngsaw-header>
</th>
</tr>
</thead>
<tbody _ngcontent-c17="">
<!-- -->
<!-- --><tr _ngcontent-c17="">
<!-- --><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>8:03 PM</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">68</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">68</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-humidity">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">100</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->%
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>SSW</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-speed">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">8</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->mph
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-speed">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->mph
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-pressure">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">29.9</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-rain">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0.0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-rain">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0.0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>Partly Cloudy</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td>
</tr><tr _ngcontent-c17="">
<!-- --><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>8:09 PM</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">69</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">69</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-humidity">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">100</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->%
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>SSW</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-speed">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">9</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->mph
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-speed">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->mph
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-pressure">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">29.9</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-rain">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0.0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-rain">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0.0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>Mostly Cloudy</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td>
</tr><tr _ngcontent-c17="">
<!-- --><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>8:51 PM</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">70</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">70</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-humidity">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">100</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->%
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>SW</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-speed">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">7</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->mph
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-speed">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->mph
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-pressure">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">29.9</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-rain">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0.0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-rain">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">0.0</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->in
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>Cloudy</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td>
</tr><tr _ngcontent-c17="">
<!-- --><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>8:53 PM</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">69</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test-true wu-unit wu-unit-temperature is-degree-visible">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">69</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->F
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-humidity">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">100</span> <span _ngcontent-c13="" class="wu-label">
<!-- -->%
    <!-- -->
</span>
<!-- -->
</span>
<!-- -->
</display-unit>
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- --><div>
<span>SW</span>
<!-- -->
</div>
<!-- -->
</ng-saw-cell-parser>
</td><td _ngcontent-c17="">
<ng-saw-cell-parser _ngcontent-c17=""><!-- -->
<!-- --><display-unit _nghost-c13=""><!-- --><span _ngcontent-c13="" class="test- wu-unit wu-unit-speed">
<!-- -->
<!-- -->
<!-- -->
<span _ngcontent-c13="" class="wu-value wu-value-to">7</span> <span _ngcontent-c13="" class="wu-label">
]

Process finished with exit code 0

Actually this is a very small snippet, since I am limited to a 30,000 character post...

Reedinationer
  • 5,661
  • 1
  • 12
  • 33
0

That website is using angular js and you know how ng-s classes work, secondly on sourceview I can't find table tag and so does bs4

dennohpeter
  • 391
  • 4
  • 16
0

You're using the wrong link, the code is sent in a dictionary file that you can easily access using the following code:

import requests

url = "https://api.weather.com:443/v1/geocode/41.28/-70.1/observations/historical.json?apiKey=6532d6454b8aa370768e63d6ba5a832e&startDate=20180729&endDate=20180729&units=e"
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0"}

all_observations = requests.get(url, headers=headers).json()["observations"] # The whole data in the table
first_observation = all_observations[0] # The first entry in the table

(PS: alternatively you can try using a headless browser as suggested by the other answers)

Nazim Kerimbekov
  • 4,712
  • 8
  • 34
  • 58