Converting nested dictionary to pandas dataframe and plotting

Question

I have a listener which is constantly logging received messages in the following way --

class Listener:

    recorded = defaultdict(lambda: defaultdict(list))

    def on_message_received(self, msg):
        self.recorded[msg.id]["timestamp"].append(msg.timestamp)
        self.recorded[msg.id]["data"].append(msg.data)

It can be assumed both the timestamp and data are floats.

For a single id, I can plot the data with --

import altair as alt
import pandas as pd
import streamlit as st

listener = Listener()

plt, frame = st.empty(), st.empty()

def plot():
    c = listener.recorded.copy()
    df = pd.DataFrame.from_dict(c[100]) # using id: 100 as an example
    chart = alt.Chart(df).mark_line().encode(x="timestamp", y="data")
    frame.write(df)
    plt.altair_chart(chart)

Which produces a dataframe and plot of:

So my question is, how would I back up a level and generate a dataframe/plot from the dictionary listener.recorded so that each unique id is plotting correctly with a legend? I cannot seem to quite get there on my own...

if you can access `c` with an index, then I believe you can loop through it and append to a dataframe during each iteration. — Vedank Pande, Apr 03 '21 at 19:18
Is something like [create pandas dataframe from dictionary of dictionaries](https://stackoverflow.com/a/33158045/15497888) or [Convert list of dictionaries to a pandas DataFrame](https://stackoverflow.com/a/53831756/15497888) what you're looking for? — Henry Ecker, Apr 03 '21 at 20:57

score 1 · Accepted Answer · answered Apr 04 '21 at 01:33

You can create the dataframe with pd.concat, reset the index, and then use Altair as normal. Here's a short example with data similar to the dictionary you are generating:

import altair as alt
import pandas as pd
import numpy as np

rng = np.random.default_rng(1701)

data = {
  100: {'timestamp': pd.date_range('2021-01-01', periods=10, freq='D'),
        'data': np.random.randint(0, 100, 10)},
  200: {'timestamp': pd.date_range('2021-01-01', periods=10, freq='D'),
        'data': np.random.randint(0, 100, 10)},
  300: {'timestamp': pd.date_range('2021-01-01', periods=10, freq='D'),
        'data': np.random.randint(0, 100, 10)},
}

df = pd.concat({k: pd.DataFrame(d) for k, d in data.items()})
df = df.reset_index(0).rename({'level_0': 'id'}, axis=1)
alt.Chart(df).mark_line().encode(
    x='timestamp:T',
    y='data:Q',
    color='id:N',
)

exactly what I was seeking. Thank you so much! – Teejay Bruno Apr 05 '21 at 01:55 — Teejay Bruno, Apr 05 '21 at 01:55

Converting nested dictionary to pandas dataframe and plotting

1 Answers1

Linked