Plotly line chart with confidence interval using groupby

Question

I'd like to plot time series simulation data as a mean with confidence intervals and compare multiple scenarios. Using the pandas groupby() and agg() functions a calculate the mean and confidence interval (upper and lower limit) see sample data (the actual data can be retrieved from my github.

Edit: I added the raw data to the and the code (as a jupyter notebook) to the git

Plotting this data for one specific parameter combination (selecting data via df = df.loc[(slice(None),1, True)]) seams simple enough:

myFig = go.Figure([
go.Scatter(
    name='Mittelwert',
    #x=df['tick'],
    y=df['mean'],
    mode='lines',
    line=dict(color='rgb(31, 119, 255)'),
),
go.Scatter(
    name='Konfidenzintervall',
    #x=df['tick'],
    y=df['ci95_hi'],
    mode='lines',
    marker=dict(color="#644"),
    line=dict(width=0),
    showlegend=True
),
go.Scatter(
    name='Konfidenzintervall',
    #x=df['tick'],
    y=df['ci95_lo'],
    marker=dict(color="#448"),
    line=dict(width=0),
    mode='lines',
    fillcolor='rgba(130, 68, 68, 0.5)',
    fill='tonexty',
    showlegend=True
)
])
myFig.update_layout(
    xaxis_title='X Achse',
    yaxis_title='Y Achse',
    title='Continuous, variable value error bars',
    hovermode="x"
    )
myFig.show()

This code gives me the that beautiful plot. The issue is a do not know how to properly plot the grouped data. When I don't select a subset all data is plotted at once. Therefore i tried to use color, facet_col and facet_row which I could get working using

px.line(reset_df, x = "tick", y="mean", color="first_factor",facet_col="second_factor")

(Since plotly apparently can't handle MultiIndex Dataframe i used reset_index() to get a DataFrame with a 'RangeIndex' first). The issue is with the latter approach I'm now missing the confidence interval and don't know how to add it (c.f. this plot).

How can I have both the ci and the grouped data within one plot? If this is not possible with pandas is it with bokeh? Thank you very much

Answer 1

( this is work in progress )

---

It's still not 100% clear to me how you'd like to display your data here. In your code sample:

px.line(reset_df, x = "tick", y="mean", color="first_factor",facet_col="second_factor")

... you're using first_factor and second_factor without saying what those are. And judging by the column names in your dataframe it could be almost any of:

['mean', 'median', 'count', 'std', 'min', 'max', 'ci95_lo', 'ci95_hi']

..or your index:

'(1.0, 0.0, False)'

But since it seems clear that you'd like to display a confidence interval around a mean across some categories, I'm guessing that this is what you're looking for:

Plot 1:

Plot 2: Zoomed in on first plot

The way to get there is full of pit-falls, and I'm not going to waste anyone's time talking about the details if this is not what you're looking for. But if it is, I'd be happy to explain everything.

Complete code:

df = pd.read_json("https://raw.githubusercontent.com/thefeinkoster/plotly_issue/main/aggregatedData.json")
df = df.reset_index()
df = pd.concat([df, df['index'].str.split(',', expand = True)], axis = 1)
df = df.rename(columns = {0:'ix', 1:'scenario', 'ci95_lo':'lo', 'ci95_hi':'hi'})

df.ix = [int(float(i[1:])) for i in df.ix]
df = df[df[2]==' True)']

dfp=pd.melt(df, id_vars=['ix', 'scenario'], value_vars=df.columns[-12:])
dfp = dfp[dfp['variable'].isin(['ix', 'mean', 'lo', 'hi'])] 

dfp = dfp.sort_values(['ix', 'variable'], ascending = False)

fig = px.line(dfp, x = 'ix', y = 'value', color = 'variable', facet_row = 'scenario')
fig.update_layout(height = 800)

fig.update_traces(name = 'interval', selector = dict(name = 'hi'))
fig.update_traces(fill = 'tonexty')
fig.update_traces(fillcolor = 'rgba(0,0,0,0)', selector = dict(name = 'mean'))
fig.update_traces(fillcolor = 'rgba(0,0,0,0)', line_color = 'rgba(44, 160, 44, 0.5)',
                  showlegend = False, selector = dict(name = 'lo'))

fig.update_yaxes(range=[0.44, 0.54])
f = fig.full_figure_for_development(warn=False)
fig.show()