Python plotting on/off data using Matplotlib

Question

I'm trying to plot data about a bunch of devices whether they're online or offline. The devices give a signal 1 when they come online and a signal 0 when they're going offline. In between, there's no data.

For just one device I use a step plot (with step=post), which works pretty well. Now I want to show by a line when one or more devices are online.

Does anyone have any tips/tricks on how to visualize this dataset? I've tried adding extra rows just before each signal to get a more continuous dataset and then plot the value of OnOff, but then I lose the categories. Do I need to convert this to a broken_barh plot? Or any other ideas?

Data:

import pandas as pd 
import matplotlib.pyplot as plt

TESTDATA = u"""\
Index;OnOff;Device
12-10-2021 10:04:04;1;device1
12-10-2021 10:04:12;0;device3
12-10-2021 10:05:05;1;device2
12-10-2021 19:05:11;0;device2
13-10-2021 05:25:17;1;device2
13-10-2021 19:26:22;0;device2
14-10-2021 15:44:44;1;device2
14-10-2021 20:54:12;0;device2
15-10-2021 04:21:42;1;device2
15-10-2021 09:15:11;0;device2
15-10-2021 17:05:05;0;device1
15-10-2021 17:05:25;1;device3
15-10-2021 17:56:45;1;device1
15-10-2021 17:57:09;1;device2
15-10-2021 21:10:20;0;device2
16-10-2021 01:51:50;1;device2
19-10-2021 10:00:13;0;device1
19-10-2021 10:04:19;0;device2
"""

df = pd.read_csv(StringIO(TESTDATA), index_col=0, sep=';', engine='python')
df.index = pd.to_datetime(df.index, format='%d-%m-%Y %H:%M:%S')
print(df)

# plot
fig, ax = plt.subplots(figsize=[16,9])

devices = list(set(df['Device']))
devices.sort(reverse=True)

for device in devices:
    ax.plot(df.index[df['Device'] == device], df['Device'][df['Device'] == device], label=device)
plt.show()

Yes, I want to be able to see at a given time which devices were online at that moment (so status 1). I thought about plotting them like my example above, on a datetime axis with a bar when the device is online. I have no idea how to convert this dataset to a broken_barh dataset, as you then need to calculate every coördinate where a bar needs to start and end. — Winnetou, Mar 16 '22 at 10:59
Can we assume that for each device, the recorded times reflect alternating states (1-0-1-0-1...) or can there be malfunctions in the registration and you have repeats (1-0-0-1-0-1-1-0....) in your database? — Mr. T, Mar 16 '22 at 11:12
Datetime objects seem difficult, but I found a thread with a figure it should somehow look like: https://stackoverflow.com/questions/24425908/matplotlib-how-to-use-timestamps-with-broken-barh. I just need to find each pair of starting and end points. If a start or endpoint is missing ideally it should be extended to the first/last datetime. — Winnetou, Mar 16 '22 at 15:01

Waldemar Walo · Answer 1 · 2022-03-16T10:02:07.487

The problem is in the ax.plot params. ax.plot requires x and y, e.g. ax.plot(x, y) your x, y are: x - df.index[df['Device'] == device] - this is correct y - df['Device'][df['Device'] == device - this is not correct

change df['Device'][df['Device'] == device to df.loc[df['Device'] == device, 'OnOff']

df.loc works by filtering rows and then columns:

df.loc[row_filter, column_filter]
row_filter = df['Device'] == device # give me all rows whre 'Device' column's value == device variable value
column_filter = 'OnOff' # give me just the OnOff column

The graph you will see may not be what you want.

You may want to replace the ax.plot with ax.step to see the below, but the data will overlap and won't be too redable:

The final solution may be to draw 3 axes, 1 for each device on shared x axis:

# plot
fig, axs = plt.subplots(3,1, figsize=[16,9], sharex=True)

devices = list(set(df['Device']))
devices.sort(reverse=True)

for device_idx, device in enumerate(devices):
    axs[device_idx].step(df.index[df['Device'] == device], df.loc[df['Device'] == device, 'OnOff'] , label=device )

Thanks for your reaction. This is more or less the solution I use for 1 device. Plotting 3 devices this way is still very well readable, however, I sometimes have up to ~30 devices and then this kind of graph gets very hard to read. — Winnetou, Mar 16 '22 at 11:01

Mr. T · Accepted Answer · 2022-03-16T17:19:37.867

Datetime objects are indeed difficult in their behavior as not all pandas/numpy/matplotlib functions accept all versions or might interpret them differently. However, we can convert datetimes into matplotlib dates, which are numerical data, making our life easier (kinda):

import pandas as pd 
import matplotlib.pyplot as plt
from matplotlib.dates import date2num

#test data generation
from io import StringIO
TESTDATA = u"""\
Index;OnOff;Device
12-10-2021 10:04:04;1;device1
12-10-2021 10:04:12;0;device3
12-10-2021 10:05:05;1;device2
12-10-2021 19:05:11;0;device2
13-10-2021 05:25:17;1;device2
13-10-2021 19:26:22;0;device2
14-10-2021 15:44:44;1;device2
14-10-2021 20:54:12;0;device2
15-10-2021 04:21:42;1;device2
15-10-2021 09:15:11;0;device2
15-10-2021 17:05:05;0;device1
15-10-2021 17:05:25;1;device3
15-10-2021 17:56:45;1;device1
15-10-2021 17:57:09;1;device2
15-10-2021 21:10:20;0;device2
16-10-2021 01:51:50;1;device2
17-10-2021 10:00:13;0;device1
19-10-2021 10:04:19;0;device2
"""
df = pd.read_csv(StringIO(TESTDATA), index_col=0, sep=';', engine='python')
df.index = pd.to_datetime(df.index, format='%d-%m-%Y %H:%M:%S')

#not necessary if presorted but we don't want to push our luck
df = df.sort_index()

fig, ax = plt.subplots(figsize=(10, 4))

#convert dates into matplotlib format
df["Mpl_date"] = df.index.map(date2num)
#group by device
gb = df.groupby("Device", sort="False")    

for pos, (_device_name, device_df) in enumerate(gb):  
    #make sure the entire datetime range is covered for each device
    prepend_df = (None, df.iloc[0].to_frame().T)[int((device_df.iloc[0] != df.iloc[0]).any())]
    append_df = (None, df.iloc[-1].to_frame().T)[int((device_df.iloc[-1] != df.iloc[-1]).any())]
    device_df = pd.concat([prepend_df, device_df, append_df])  
    device_df["OnOff"].iloc[[0, -1]] = 1 - device_df["OnOff"].iloc[[1, -2]] 
    #calculate time differences as broken_barh expects a list of tuples (x_start, x_width)
    device_df["Mpl_diff"] = -device_df["Mpl_date"].diff(-1)

    #and plot each broken barh, starting with the first status 1
    ax.broken_barh(list(zip(device_df["Mpl_date"], device_df["Mpl_diff"]))[1- device_df["OnOff"].iloc[0]::2], (pos-0.1, 0.2))

ax.set_yticks(range(gb.ngroups))
ax.set_yticklabels(gb.groups.keys()) 
ax.xaxis_date()
ax.set_xlabel("Date")
plt.tight_layout()       
plt.show()

Sample output:

Most of the code is just necessary to take care of cases where the status at the beginning or end is not explicitly declared by the original dataframe. That the status is 1 and 0, however, makes the coding easier as it can be directly translated into indexes.

P.S. The first bar of device 3 is visible in the original plot but not in the downsampled image stored here on SO.

Is `Keyword1` correct and of dtype integer? The code assumes and takes advantage of the fact that `OnOff` is an integer 0 or 1, and this should be true for Keyword 1, too. Maybe it is a float value or string? Then you should convert it into `1-int(device_df["Keyword1"].iloc[0])`. This line creates a list of tuples (x_start, x_width), takes every second, and makes sure that it starts with the first status 1 to plot the on status. The list indexing method `[(a0, b0), (a1, b1),...,(an, bn)][start: stop: step]` expects naturally only integers for start. — Mr. T, Mar 17 '22 at 08:21
If it is a float value, then I would also check that the column does not contain NaN values. Such data irregularities are not considered as a possibility because you said you have checked that the status values alternate in each category. — Mr. T, Mar 17 '22 at 08:23
Thank for all very helpful hints! The dtype of Keyword1 indeed was float, although it only contains 0 or 1. Maybe because the original dataframe contains NaN values, but they are removed before. Changing it to int makes your code work exactly the desired way! :) P.S. I accidentally removed my first comment. Below is the error message I got after using my original dataset with float values: ax.broken_barh(list(zip(device_df["Mpl_date"], device_df["Mpl_diff"]))[1- device_df["Keyword1"].iloc[0]::2], (pos-0.1, 0.2)) TypeError: slice indices must be integers or None or have an __index__ method — Winnetou, Mar 17 '22 at 08:40

score 0 · Answer 3 · answered Mar 16 '22 at 18:27

# first I would assume that all devices have to start from the unknow state, instead of assuming they are off, 
# thus lets add one row at the begining
new_index_first_element = df.index[0]-pd.Timedelta(seconds=1)
new_index = [new_index_first_element] + df.index.to_list()

devices = sorted(df.Device.unique())

# lets create a new dataframe where each device will have its own column and
# each entry will track the state of each device
df2 = pd.DataFrame(index = new_index, columns=devices) 

for i_iloc in range(1,len(df2)): # i have to be able to reffer to previous row, thus I will go with iloc, instead of loc
    # first copy previous status of all devices to current row
    df2.iloc[i_iloc] = df2.iloc[i_iloc-1]
    
    # now lets update the status for device that changed
    current_row_idx = df2.iloc[[i_iloc]].index
    device_to_update = df.loc[current_row_idx, 'Device']
    status_to_update = df.loc[current_row_idx, 'OnOff']
    df2.at[current_row_idx, device_to_update] = status_to_update

df2

This is how the DF will look like, it has an additional row with NaNs as we do not know what the status of those devices are.

# and plot
fig, ax = plt.subplots(figsize=[16,9])
df2.plot(kind='bar', stacked=True, color=['red', 'skyblue', 'green'], ax=ax)

I dont think that plotting a 'broken_barh plot' will do a good job here, this stacked barplot will be way better.

Python plotting on/off data using Matplotlib

3 Answers3