0

I want to process "Burst" data- a time series with bursts. The data can be pretty noisy. I am really only interested in the burst duration but my burst detection algorithm only really works if the data has no slope. Now my question is : How do i find a linear slope for this type of data without doing it manually? My main problem is that there can be burst which exceed either end of my time(x) axis. Otherwise i could probably just find the mean of the first and last 20 datapoints and fit a linear function. Basically i want to find the red line in the follwing picture and subtract it. I guess a linear regression either through the burst or the sloped baseline would do the trick but i am somehow stuck.

enter image description here

Code:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from matplotlib import rcParams,pylab 
from matplotlib.patches import Rectangle
import random

#set plot properties
sns.set_style("white")
rcParams['font.size'] = 14
FIG_SIZE=(12, 15)
# Simulate data
timepoints = 4000
r = pd.Series(np.floor(np.ones(timepoints)*20 + np.random.normal(scale=10, 
size=timepoints)))   #target events
r[r<0] = 0 #set negative values to 0
# #add some bursts to the data
heights = [35,45,50,55,40,60,70]
starts =[100,300,700,950,1200,1800,2100,2550,2800,3100,3500,3800]
ends=[200,400,800,1100,1500,1900,2400,2625,2950,3350,3700,4000]
for x,y in zip(starts,ends):
    r[x:y] = r[x:y] + random.choice(heights) #+ np.random.normal(scale=10, size=200)
# add linear slope to data
slope=0.02
linear_slope=np.arange(timepoints) *slope
burst_data_with_slope= r + linear_slope
#Fig setup
fig, (ax1, ax2) = plt.subplots(2, figsize=FIG_SIZE,sharey=False)
#
ax1.set_ylabel('proportion of target events', size=14)
ax1.set_xlabel('time (sec)', size=14)
ax2.set_xlabel('time (sec)', size=14)
ax2.set_ylabel('proportion of target events', size=14)
ax1.set_xlim([0, timepoints])
ax2.set_xlim([0, timepoints])
ax1.plot(burst_data_with_slope, color='#00bbcc', linewidth=1)
ax1.set_title('Original Data', size=14)
ax2.plot(burst_data_with_slope, color='#00bbcc', linewidth=1)
ax2.plot(linear_slope, color='red', linewidth=1)
ax2.set_title('Original Data substracted slope', size=14)



# Finaly plot
plt.subplots_adjust(hspace=0.5)
plt.show()

Thanks in advance !

NorrinRadd
  • 545
  • 6
  • 21
  • 1
    The linear regression keeps a small trend as the height of the bursts are not constant... maybe there is a better solution – xdze2 Sep 01 '18 at 21:27
  • maybe you will have better chance on https://stats.stackexchange.com/... (it is actually a wide subject, see [step detection](https://en.wikipedia.org/wiki/Step_detection)) Did you try the simple linear regression? isn't it good enough? Or you could iterate: detect the burst on the data corrected with a first linear regression, then perform a second regression only on the "non-burst" points (the base line), then perform the burst detection again... – xdze2 Sep 04 '18 at 12:02

0 Answers0