Detecting Pattern in Real Time Data array in Python

Question

I'm trying to detect specific pattern in the Real time data (Time Series). For the visualization, I'll show the data in two parts here.

Pattern: I'm trying to search for in time series,

DataWindow: data buffer(window) I slide in real time to keep track of history.

Here is my recorded data(red boxes shows the pattern that I want to detect), but this can be different since it is Real Time:

The above data doesn't have a lot of noise (at least for this collection) - as far as I look at the resolutions, peaks (maybe I would say sinusoidal peaks) are distinguishable at first glance. That is why applying a moving average filter does not help me at all.

The below image shows some samples from real time data but in the saved data, plotter applies extrapolation to draw continous plot. In general, data samples look like the image below or maybe with more resolution than this image.

For the initial start, I've tried Spike Detection in a Time-Seriesusing moving average and did not work as I expected. I've also tried some solutions here from this thread Detecting patterns from two arrays of data in Python and the results are not good enough for me to raise a flag in the patterns during run-time(there are many false positives)

Also, as you might realize from the saved real time data that, patterns can have different scale and most importantly can have different offset. That is the problem I guess for me to apply above solutions on my problem to get distinguishable results.

To give some example to try out, these can be used for the Pattern and DataWindow Pattern = [5.9, 5.6, 4.08, 2.57, 2.78, 4.78, 7.3, 7.98, 4.81, 5.57, 4.7]

SampleTarget = [4.74, 4.693, 4.599, 4.444, 3.448, 2.631, 1.845, 2.032, 2.415, 3.714, 5.184, 5.82, 5.61, 4.841, 3.802, 3.11]

SampleTarget2 = [5.898, 5.91, 5.62, 5.25, 4.72, 4.09, 3.445, 2.91, 2.7, 2.44, 2.515, 2.79, 3.25, 3.915,4.72, 5.65, 6.28, 7.15, 7.81, 8.2, 7.9, 7.71, 7.32, 6.88, 6.44, 6.0,5.58, 5.185, 4.88, 4.72, 4.69, 4.82]

I am trying to solve this problem on Python for PoC. UPDATE: Dataset is added, includes first two red boxes and a bit wider side as well, which is shown in the saved real time data.dataset

Answer 1

You can compute the gradient of the data and use a threshold to identify the features. Here I use a triple mask to get the down/up/down feature.

I commented the code to give you the main steps, so I hope it is comprehensive.

import pandas as pd
import matplotlib.pyplot as plt

# read data
s = pd.read_csv('sin_peaks.txt', header=None)[0]
# 0    5.574537
# 1    5.736071
# 2    5.965132
# 3    6.164344
# 4    6.172413

thresh = 0.5 # threshold of derivative
span = 10    # max span of the feature (in number of points)

# calculate gradient
# if the points are not evenly spaced
# you should also divide by the spacing
s2 = s.diff()

# get points outside of threshold
m1 = s2.lt(-thresh)
m2 = s2.gt(thresh)

# extend masks
m1_fw = m1.where(m1).ffill(limit=span)
m1_bw = m1.where(m1).bfill(limit=span)
m2_fbw = m2.where(m2).ffill(limit=span).bfill(limit=span)

# slice data where all conditions are met
# up peak & down peak in the "span" before and down peak in the "span" after
peaks = s[m1_fw & m1_bw & m2_fbw]

# group peaks
groups = peaks.index.to_series().diff().ne(1).cumsum()

# plot identified features
ax = s.plot(label='data')
s.diff().plot(ax=ax, label='gradient')
ax.legend()

ax.axhline(thresh, ls=':', c='k')
ax.axhline(-thresh, ls=':', c='k')

for _, group in peaks.groupby(groups):
    start = group.index[0]
    stop = group.index[-1]
    ax.axvspan(start, stop, color='k', alpha=0.1)