0

I have some data ( a timeseries of some reponse time) and i am trying to detect a real change in the signal, as opposed to a noisy change.

For example, the below chart:

time series

You can see it is extremely noisy, but visually i would say there are 3 points that i would like to highlight as changes (marked in yellow)

I am currently using python ruptures, in particular the Pelt algorithm (Although i have tried the binary search as well as the window search)

I need the model to be flexibile enough so that it works on other noisy data, without giving false positives, for example, on the below, i would want no Change point detected...

enter image description here

I have been testing hte ruptures package, and it sort of works but delivers many false positives. Would it make sense to try and smooth the outliers in the data before applying a CPD algorithm?

Im just shooting ideas out... happy for any input, it would be greatly appreciated

Alex Walton
  • 173
  • 3
  • 17
  • Would a generic sliding-window median filter work? For example, something like [scipy.ndimage.median_filter](https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.median_filter.html) as suggested in [this answer](https://stackoverflow.com/questions/37671432/how-to-calculate-running-median-efficiently)? – Joshua Voskamp Dec 13 '22 at 20:15

1 Answers1

1

My idea is to use moving median/mean to find the peak. We can use numpy to reach the goal. Here is the roughly ideas. 1. set a small window to calculate the moving average. Use the convolve will be fast, the kernal array size will be the winow size. data = np.convolve(datax, self.kernal, 'valid') 2. find the start and end time of event which are larger than 5 time stdev with you moving average point. condition = data_raw > (date + 5*stdev) event_start = np.where(np.any(np.logical_and(~condition[0:-1], condition[1:])))[0] event_end = np.where(np.any(np.logical_and(condition[0:-1], ~condition[1:])))[0] 3. apply some roles to exclude the small peak, you can use for loop to check every event. this easy program should be very fast we used for high sampling rate in real-time analysis.of course you can use median filter in scipy to instead the first step's convolve mean.

yuanjie li
  • 11
  • 2
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 12 '22 at 09:35
  • Please consider adding specific library suggestions, or linking to an existing implementation of a sliding-window median. – Joshua Voskamp Dec 13 '22 at 20:18