realtime plotting pandas dataframe

Question

I am new to matplotlib and trying to display a real time plot of the last hour's data for three variables that I'm downloading from an api via my function read_API(). The data is in a pandas dataframe with a DateTimeIndex. For example:

In: dframe.head()
Out:
                                 A          B         C
timestamp                                                            
2017-05-11 16:21:55        0.724931  0.361333   0.517720  
2017-05-11 16:22:25        0.725386  0.360833   0.518632
2017-05-11 16:22:55        0.725057  0.361333   0.521157
2017-05-11 16:23:25        0.724402  0.362133   0.520002

The simplified code is:

import pandas as pd
import matplotlib.pyplot as plt
import datetime as dt
while True:
    dframe = read_API()
    dframe['timestamp'] = dframe['timestamp'] + pd.DateOffset(hours=timezone)
    dframe = dframe.set_index('timestamp')
    end = dframe.index.max()
    start= end.to_datetime() - dt.timedelta(hours=1)
    dframe = dframe.loc[start:end]
    plt.ion()
    fig, ax = plt.subplots()
    plt.pause(0.0001)
    ax.plot_date(dframe.index.to_pydatetime(), dframe,marker='', linestyle='solid')
    plt.draw()

It is producing updated plots avery few seconds , but: 1) each plot apepars in a new window (called Figure 1, Figure 2, Figure 3.....). I want a single window with the plot overwriting the previous one 2) When each plot appears, it is blank. Then another blank one appears, then another, then the first one is completed, and so on. The actual plotting lags about 3 figures behind..... I am a bit confused about the distinction between plots and subplots, and think the problem may be related to this.

Answer 1

I think the issue with your code is that you invoke fig, ax = plt.subplots() each time you refresh your data. This creates a new Figure each time, and thus you see new frames pop up.

Instead, you want to create the Figure outside of your while loop, and only refresh the Axes once you loaded new data.

I've used the basic example you've provided to create a self-updating Figure.

import pandas as pd
import matplotlib.pyplot as plt 
import datetime as dt

data = [ 
    {'timestamp': '2017-05-11 16:21:55', 'A': 0.724931, 'B': 0.361333, 'C': 0.517720},
    {'timestamp': '2017-05-11 16:22:25', 'A': 0.725386, 'B': 0.360833, 'C': 0.518632},
    {'timestamp': '2017-05-11 16:22:55', 'A': 0.725057, 'B': 0.361333, 'C': 0.521157},
    {'timestamp': '2017-05-11 16:23:25', 'A': 0.724402, 'B': 0.362133, 'C': 0.520002},
]
df = pd.DataFrame(data) 
df.set_index("timestamp")

plt.ion()
fig, ax = plt.subplots()
while True:
    dframe = df.copy()
    dframe['timestamp'] = pd.to_datetime(dframe['timestamp']) + pd.DateOffset(hours=2)
    dframe = dframe.set_index('timestamp')
    end = dframe.index.max()
    start= end.to_datetime() - dt.timedelta(hours=1)
    dframe = dframe.loc[start:end]
    plt.pause(0.0001)
    ax.plot_date(dframe.index.to_pydatetime(), dframe, marker='', linestyle='solid')

---

Edit 1

I cannot reproduce the raised Warning, but my guess would be that it's related to the pause call. Maybe try swapping the following and editing the pause time.

   ax.plot_date(dframe.index.to_pydatetime(), dframe, marker='', linestyle='solid')
   plt.pause(0.01)

---

Edit 2

Fixing the colours is quite simple. Define your colour palette and then pick from it.

colors = ['r', 'g', 'b']

plt.ion()
fig, ax = plt.subplots()
while True:
    dframe = df.copy()
    # Your data manipulation
    # ...
    dframe = dframe.loc[start:end]
    for i, column in enumerate(dframe.columns):
        ax.plot(dframe.index.to_pydatetime(), dframe[column], color=colors[i], marker=None, linestyle='solid')   
    plt.pause(0.1)

If you have more columns, add more colours to the colors array. Alternatively, generate it on-the-fly depending on the number of columns in dframe.