Rolling Average to calculate rainfall intensity

Question

I have some real rainfall data recorded as the date and time, and the accumulated number of tips on a tipping bucket rain-gauge. The tipping bucket represents 0.5mm of rainfall. I want to cycle through the file and determine the variation in intensity (rainfall/time) So I need a rolling average over multiple fixed time frames: So I want to accumulate rainfall, until 5minutes of rain is accumulated and determine the intensity in mm/hour. So if 3mm is recorded in 5min it is equal to 3/5*60 = 36mm/hr. the same rainfall over 10 minutes would be 18mm/hr...

So if I have rainfall over several hours I may need to review at several standard intervals of say: 5, 10,15,20,25,30,45,60 minutes etc... Also the data is recorded in reverse order in the raw file, so the earliest time is at the end of the file and the later and last time step appears first after a header: Looks like... (here 975 - 961 = 14 tips = 7mm of rainfall) average intensity 1.4mm/hr But between 16:27 and 16:34 967-961 = 6 tips = 3mm in 7 min = 27.71mm/hour

7424 Figtree (O'Briens Rd)
DATE     :hh:mm Accum Tips
8/11/2011 20:33     975
8/11/2011 20:14     974
8/11/2011 20:04     973
8/11/2011 20:00     972
8/11/2011 19:35     971
8/11/2011 18:29     969
8/11/2011 16:44     968
8/11/2011 16:34     967
8/11/2011 16:33     966
8/11/2011 16:32     965
8/11/2011 16:28     963
8/11/2011 16:27     962
8/11/2011 15:30     961

Any suggestions?

Surely that's 967 - **962** = 5 tips = 2.5mm in between 16:27 and 16:34? — Lauritz V. Thaulow, Nov 28 '11 at 12:20

noio · Answer 1 · 2011-11-28T11:35:57.940

I am not entirely sure what it is that you have a question about.

Do you know how to read out the file? You can do something like:

data = [] # Empty list of counts

# Skip the header
lines = [line.strip() for line in open('data.txt')][2::]

for line in lines:
    print line
    date, hour, count = line.split()
    h,m = hour.split(':')
    t = int(h) * 60 + int(m)      # Compute total minutes
    data.append( (t, int(count) ) ) # Append as tuple

data.reverse()

Since your data is cumulative, you need to subtract each two entries, this is where python's list comprehensions are really nice.

data = [(t1, d2 - d1) for ((t1,d1), (t2, d2)) in zip(data, data[1:])]
print data

Now we need to loop through and see how many entries are within the last x minutes.

timewindow = 10
for i, (t, count) in enumerate(data):
    # Find the entries that happened within the last [...] minutes
    withinwindow = filter( lambda x: x[0] > t - timewindow, data )
    # now you can print out any kind of stats about this "within window" entries
    print sum( count for (t, count) in withinwindow )

`zip` matches up items from two or more lists. So `zip([1,2],['a','b']) = [(1,'a'), (2,'b')]`. Using it this way, tuples are returned consisting of consecutive items, since data[1:] starts from the second entry. — noio, Dec 01 '11 at 10:54

score 0 · Answer 2 · edited May 23 '17 at 10:27

0

Since the time stamps do not come at regular intervals, you should use interpolating to get the most accurate results. This will make the rolling average easier too. I'm using the Interpolate class in this answer in the below code.

from time import strptime, mktime

totime = lambda x: int(mktime(strptime(x, "%d/%m/%Y %H:%M")))
with open("my_file.txt", "r") as myfile:
    # Skip header
    for line in myfile:
        if line.startswith("DATE"):
            break
    times = []
    values = []
    for line in myfile:
        date, time, value = line.split()
        times.append(totime(" ".join((date, time))))
        values.append(int(value))
times.reverse()
values.reverse()
i = Interpolate(times, values)

Now it's just a matter of choosing your intervals and computing the difference between the endpoints of each interval. Let's create a generator function for that:

def rolling_avg(cumulative_lookup, start, stop, step_size, window_size):
    for t in range(start + window_size, stop, step_size):
        total = cumulative_lookup[t] - cumulative_lookup[t - window_size]
        yield total / window_size

Below I'm printing the number of tips per hour in the previous hour with 10 minute intervals:

start = totime("8/11/2011 15:30")
stop = totime("8/11/2011 20:33")
for avg in rolling_avg(i, start, stop, 600, 3600):
    print avg * 3600

EDIT: Made totime return an int and created the rolling_avg generator.

edited May 23 '17 at 10:27

Community

1
1

answered Nov 28 '11 at 11:40

Lauritz V. Thaulow

49,139
12
73
92

This is great thank you.... What if I need to create rolling averages of many intervals? Say for every minute 1 minute to 3600 ? – Rudy Van Drie Mar 25 '12 at 22:00
I'm afraid I don't quite understand what you're asking. For a rolling average there are only four parameters: start, stop, the step size, which is 600 in the above example, and the window size, which is 3600 in the above example. If what you wish is to look at many different window sizes or step sizes, you may create a `rolling_avg` generator for each window size or step size and iterate over each of them. Does that answer your question? If not, can you give a concrete example of something you wish to do? – Lauritz V. Thaulow Mar 26 '12 at 11:48
Yes I think it does... I will feed it a list of Windows. – Rudy Van Drie Mar 30 '12 at 10:03
What I am trying to do is create a list of Rainfall intensities (mm/hr) for a set of standard Durations 1 minute to upto 72 Hours, for storm events such as those that occurred in Queensland in 2011, that lasted for a week or so... – Rudy Van Drie Mar 30 '12 at 10:10
@Rudy In that case, creating one rolling_avg generator for each window size in a list of standard durations (in seconds) and then iterating over each generator sounds correct. – Lauritz V. Thaulow Mar 30 '12 at 10:52

Rolling Average to calculate rainfall intensity

2 Answers2

Linked