Creating a dataframe where one of the arrays has a different length

Question

I am learning to scrape data from website through Python. Extracting weather information about San Francisco from this page. I get stuck while combining data into a Pandas Dataframe. Is it possible to create a dataframe where each rows have different length?

I have already tried 2 ways based on answers here, but they are not excatly what I am looking for. Both answers shift the values of temps column to up. Here is the screen what I try to explain..

1st way: https://stackoverflow.com/a/40442094/10179259

2nd way: https://stackoverflow.com/a/19736406/10179259

import requests
from bs4 import BeautifulSoup
import pandas as pd


page = requests.get("http://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168")

soup = BeautifulSoup(page.content, 'html.parser')

seven_day = soup.find(id="seven-day-forecast")

forecast_items = seven_day.find_all(class_="tombstone-container")

periods=[pt.get_text() for pt in seven_day.select('.tombstone-container .period-name')]

short_descs=[sd.get_text() for sd in seven_day.select('.tombstone-container .short-desc')]

temps=[t.get_text() for t in seven_day.select('.tombstone-container .temp')]

descs = [d['alt'] for d in seven_day.select('.tombstone-container img')]


#print(len(periods), len(short_descs), len(temps), len(descs))

weather = pd.DataFrame({
        "period": periods, #length is 9
        "short_desc": short_descs, #length is 9
        "temp": temps, #problem here length is 8
        #"desc":descs #length is 9
    })


print(weather)

I expect that first row of the temp column to be Nan. Thank you.

Answer 1

You can loop each forecast_items value with iter and next for select first value, if not exist is assigned fo dictionary NaN value:

page = requests.get("http://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168")

soup = BeautifulSoup(page.content, 'html.parser')

seven_day = soup.find(id="seven-day-forecast")

forecast_items = seven_day.find_all(class_="tombstone-container")

out = []
for x in forecast_items:
    periods = next(iter([t.get_text() for t in x.select('.period-name')]), np.nan)
    short_descs = next(iter([t.get_text() for t in x.select('.short-desc')]), np.nan)
    temps = next(iter([t.get_text() for t in x.select('.temp')]), np.nan)
    descs = next(iter([d['alt'] for d in x.select('img')]), np.nan)
    out.append({'period':periods, 'short_desc':short_descs, 'temp':temps, 'descs':descs})

---

weather = pd.DataFrame(out)
print (weather)
                                               descs               period  \
0                                                     NOW until4:00pm Sat   
1  Today: Showers, with thunderstorms also possib...                Today   
2  Tonight: Showers likely and possibly a thunder...              Tonight   
3  Sunday: A chance of showers before 11am, then ...               Sunday   
4  Sunday Night: Rain before 11pm, then a chance ...          SundayNight   
5  Monday: A 40 percent chance of showers.  Cloud...               Monday   
6  Monday Night: A 30 percent chance of showers. ...          MondayNight   
7  Tuesday: A 50 percent chance of rain.  Cloudy,...              Tuesday   
8  Tuesday Night: Rain.  Cloudy, with a low aroun...         TuesdayNight   

                               short_desc         temp  
0                           Wind Advisory          NaN  
1                       Showers andBreezy  High: 56 °F  
2                           ShowersLikely   Low: 49 °F  
3                     Heavy Rainand Windy  High: 56 °F  
4  Heavy Rainand Breezythen ChanceShowers   Low: 52 °F  
5                           ChanceShowers  High: 58 °F  
6                           ChanceShowers   Low: 53 °F  
7                             Chance Rain  High: 59 °F  
8                                    Rain   Low: 53 °F