Real time highpass filter

Question

Hi Implemented a highpass filter following the this. But, my data is not predefined and it comes every in real-time. I am not sure if this correct.

def sine_generator(fs, sinefreq, duration):
    T = duration
    nsamples = int(fs * T)
    w = 2. * np.pi * sinefreq
    t_sine = np.linspace(0, T, nsamples, endpoint=False)
    y_sine = np.sin(w * t_sine)
    result = pd.DataFrame({ 
        'sine' : y_sine} ,index=t_sine)
    return result

def butter_highpass(cutoff, fs, order=5):
    nyq = 0.5 * fs
    normal_cutoff = cutoff / nyq
    print("Filter order is ={}, Normalized cutoff ={}".format(order, normal_cutoff))
    b, a = signal.butter(order, normal_cutoff, btype='high', analog=False)
    print(b,a)
    return b, a


# signal generation
sampling_freq = int(1e5)
cutoff_freq = 100/2*np.pi
duration = .01
data = sine_generator(sampling_freq, 10, duration) 
data['sine'] = 0.1*data['sine'] + 0.2*sine_generator(sampling_freq, int(1e3), duration)['sine'] + 0.2


t = 7 # signal.filtfilt requires atleast t values to filter. t increases as the order increases
b, a = butter_highpass(cutoff_freq, sampling_freq, order=1)
for i in range(np.shape(data.sine.values)[0]-t):
    d = data.sine.values[i:i+t+1]
    y = signal.filtfilt(b, a, d)
    print(y)

I think this is not the rightway.

Well, you can't filter one element at a time. Even with real-time values, you need to bundle up a set of values to do processing. You do the filtering on that set of values after you have collected it. — Tim Roberts, Apr 26 '21 at 18:40
You aren't going to get very much high-frequency content in a sample 7 elements long. — Tim Roberts, Apr 26 '21 at 18:42
Should I use all the previous data, like `data.sine.values[:i+t+1]`? Is there a better implementation of where it updates internally with the old and the new data? — kosa, Apr 26 '21 at 18:48
@TimRoberts I understand that it is not possible to filter individual elements. But I was hoping it would update itself from the old data and gets better at filtering, over time. — kosa, Apr 26 '21 at 18:50
"Gets better?" No, it is deterministic. A given input always produces the same output. There's no state. I don't understand your numbers, either. Your data is 10kHz, and you're doing a high-pass filter with a cutoff of about 8 Hz. You won't even see any 8Hz data until you get more than 1,200 samples. — Tim Roberts, Apr 26 '21 at 18:54
This is an MWE, I want to remove the dc value from the data and lower frequency values. — kosa, Apr 26 '21 at 18:56
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/231623/discussion-between-kosa-and-tim-roberts). — kosa, Apr 26 '21 at 19:02
Arent filters essentially Ordinary Differential Equations or Discrete Difference equations? So What they need are the initial conditions to filter the data? — kosa, Apr 26 '21 at 19:04
But it's not long-term. It's a bilinear transform, essentially a small-window convolution. A 5-th order Butterworth doesn't look behind more than its order. https://en.wikipedia.org/wiki/Butterworth_filter — Tim Roberts, Apr 26 '21 at 19:12

score 0 · Answer 1 · answered Apr 26 '21 at 19:20

If you want to filter the data that is coming, you must apply the differences equation with the filter coefficients using samples in the past and present.

See details: https://www.dsprelated.com/freebooks/filters/Difference_Equation_I.html

But if you already had this saved data into a buffer-array, then you just signal.lfilter(b,a,input_data) in python.

score 0 · Answer 2 · answered Apr 26 '21 at 19:20

Let's look at the data. Here, I plot your entire sine wave data set, plus the result of the high-pass filtering:

import numpy as np
from scipy import signal
import matplotlib.pyplot as plt

def sine_generator(fs, sinefreq, duration):
    T = duration
    nsamples = int(fs * T)
    w = 2. * np.pi * sinefreq
    t_sine = np.linspace(0, T, nsamples, endpoint=False)
    y_sine = np.sin(w * t_sine)
    return y_sine

def butter_highpass(cutoff, fs, order=5):
    nyq = 0.5 * fs
    normal_cutoff = cutoff / nyq
    print("Filter order is ={}, Normalized cutoff ={}".format(order, normal_cutoff))
    b, a = signal.butter(order, normal_cutoff, btype='high', analog=False)
    print(b,a)
    return b, a


# signal generation
sampling_freq = int(1e5)
cutoff_freq = 100/2*np.pi
duration = .01
data = sine_generator(sampling_freq, 10, duration) 
data = 0.1*data + 0.2*sine_generator(sampling_freq, int(1e3), duration) + 0.2

plt.plot(data)

t = 7 # signal.filtfilt requires atleast t values to filter. t increases as the order increases
b, a = butter_highpass(cutoff_freq, sampling_freq, order=1)
result = signal.filtfilt( b, a, data )
plt.plot(result)
plt.show()

And here is the result, original in blue, filtered in yellow, showing that the DC is filtered out. You just need to provide enough samples to have the filtering take effect:

And here is using lfilter instead of filtfilt:

Real time highpass filter

2 Answers2