0

I have tried to implement an high-pass filter in python to hourly precipitation measurements. I need to filter low frequency over 24 hour.

Data

I have modified this code from: Python High Pass Filter

import numpy as np
import pandas as pd
from pandas import Series
import csv
import matplotlib.pyplot as plt
from scipy import signal

def butter_highpass(cutoff, fs, order=5):
    nyq = 0.5 * fs
    normal_cutoff = cutoff / nyq
    b, a = butter(order, normal_cutoff, btype='high', analog=False)
    return b, a

def butter_highpass_filter(data, cutoff, fs, order=5):
    b, a = butter_highpass(cutoff, fs, order=order)
    y = filtfilt(b, a, data)
    return y

df= pd.read_csv('pp_2010.csv', sep=',', parse_dates=['Date'])
df= df.set_index('Date')

dt=3600
fs=1/float(dt)
cutoff=1/float(dt*24)
data=np.array(df['pp/hr'])
data_fl = butter_highpass_filter(data,cutoff,fs)

plt.plot(data,label='obs')
plt.plot(data_fl,label='filtered')
plt.show()

The code is working, I have tried it with the example of the link. In my case data_fl is an array of nan's.

I think the problem is because is non-sinusoidal, but I am not sure. If this is the problem, How could I filter this dataset?

Any hints what is the mistake?

thanks

Fco Lang
  • 1
  • 2
  • What "did not work"? Was there an error when you ran this code? if so, which line, what was the full error message? Or did the code run without errors but the output wasn't what you expected? If so, what was the output, how did it differ from what you expected? And please edit this information into the question. – DisappointedByUnaccountableMod Jun 21 '17 at 09:00
  • No error messages, just the output is an array of nan's. – Fco Lang Jun 21 '17 at 09:38
  • Have you tried giving the filter some known data with known output? Please add working code to your question, for loading the file and producing the output file, and with your choices for cutoff, fs and any other needed variables, i.e. everything needed to run this filter against your data and produce the same output. Looking for a Minimal Complete Verifiable Example https://stackoverflow.com/help/mcve – DisappointedByUnaccountableMod Jun 21 '17 at 10:19
  • Just realised your data is already high-pass filtered (compared to 24h period) because each hour sample is independent of the previous hours. So, what were you hoping to get by additional high-pass filtering? – DisappointedByUnaccountableMod Jun 21 '17 at 11:18
  • I have changed it for a working code. The data is not filtered. It's real measurements direct from a meteorological station. Theoretically, there are long-term processes that affect the data, and that those processes of scale greater than 24 hours that I want to filter. As it's precipitation every hour is independent, I guess this is the reason due to it's not working. – Fco Lang Jun 21 '17 at 11:34
  • OK it is raw data, what I mean is that the sampling has already effectively high-pass filtered. For example there is no 0Hz component in your samples. And as the data is one-sided (never goes negative) that may also be a problem for any attempt at filtering. You can't make a sample negative, so if you were to attempt to "filter out" something, I think you will have to do something with your data to ensure it is (almost) never zero. For example you could easily work out daily/weekly/10-day rainfall on a rolling (e.g. every hour, the sum of last 24 hour samples) or period basis, and filter that. – DisappointedByUnaccountableMod Jun 21 '17 at 12:45
  • Your samples are deltas (the rate, mm/hour) i.e. the piecewise differential of the rainfall. A differentiator is a high-pass filter. – DisappointedByUnaccountableMod Jun 21 '17 at 12:48
  • I understand. Thanks for your help! – Fco Lang Jun 22 '17 at 06:01

0 Answers0