Pandas finding local max and min

Question

I have a pandas data frame with two columns one is temperature the other is time.

I would like to make third and fourth columns called min and max. Each of these columns would be filled with nan's except where there is a local min or max, then it would have the value of that extrema.

Here is a sample of what the data looks like, essentially I am trying to identify all the peaks and low points in the figure.

Are there any built in tools with pandas that can accomplish this?

Should the result be robust against noise? Otherwise, you could just compare the values of the Series to its shifts. — fuglede, Dec 29 '17 at 14:24
I'm not worried about noise in this case, if it were a noisy signal I would just filter then look for max/min on the filter result — Mustard Tiger, Dec 29 '17 at 14:27
You could alternatively fit a very simple (e.g. linear with one or two covariates) model to the data, and then from the residual terms keep those whose deviations are in the `q`% smallest or largest categories, using [pd.quantile](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.quantile.html). — Nelewout, Dec 29 '17 at 14:50

Foad S. Farimani · Answer 1 · 2020-12-15T14:51:48.317

123

The solution offered by fuglede is great but if your data is very noisy (like the one in the picture) you will end up with lots of misleading local extremes. I suggest that you use scipy.signal.argrelextrema() method. The .argrelextrema() method has its own limitations but it has a useful feature where you can specify the number of points to be compared, kind of like a noise filtering algorithm. for example:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from scipy.signal import argrelextrema

# Generate a noisy AR(1) sample

np.random.seed(0)
rs = np.random.randn(200)
xs = [0]
for r in rs:
    xs.append(xs[-1] * 0.9 + r)
df = pd.DataFrame(xs, columns=['data'])

n = 5  # number of points to be checked before and after

# Find local peaks

df['min'] = df.iloc[argrelextrema(df.data.values, np.less_equal,
                    order=n)[0]]['data']
df['max'] = df.iloc[argrelextrema(df.data.values, np.greater_equal,
                    order=n)[0]]['data']

# Plot results

plt.scatter(df.index, df['min'], c='r')
plt.scatter(df.index, df['max'], c='g')
plt.plot(df.index, df['data'])
plt.show()

Some points:

you might need to check the points afterward to ensure there are no twine points very close to each other.
you can play with n to filter the noisy points
argrelextrema returns a tuple and the [0] at the end extracts a numpy array

edited Dec 15 '20 at 14:51

answered Jun 13 '18 at 11:44

Foad S. Farimani

12,396
15
78
193

11

This is good solution. I wrote a small blogpost about it: http://eddwardo.github.io/pandas/timeseries/2019/06/05/finding-local-extreams-in-pandas-time-series/ – eddd Jun 05 '19 at 18:14
2

Excellent blog post @eddd , that really helped me understand it! – Rob H Nov 11 '19 at 14:08
2

@eddd the page is down – Foad S. Farimani Dec 15 '20 at 14:47
6

@Foad https://eddwardo.github.io/posts/2019-06-05-finding-local-extreams-in-pandas-time-series/ – eddd Dec 17 '20 at 23:53
2

The best solution as well as the fastest. Did not know about `argrelextrema` – linello Sep 25 '21 at 21:37
1

Great idea, but note that this solution seems to have issues when minima/maxima are numerically identical. For instance, `np.less_equal` could result in detecting them all, whereas `np.less` could result in not detecting them at all. See [this question](https://stackoverflow.com/questions/33937445/argrelextrema-and-flat-extrema). – bluenote10 Dec 30 '22 at 09:53

fuglede · Accepted Answer · 2017-12-29T17:04:11.467

55

Assuming that the column of interest is labelled data, one solution would be

df['min'] = df.data[(df.data.shift(1) > df.data) & (df.data.shift(-1) > df.data)]
df['max'] = df.data[(df.data.shift(1) < df.data) & (df.data.shift(-1) < df.data)]

For example:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Generate a noisy AR(1) sample
np.random.seed(0)
rs = np.random.randn(200)
xs = [0]
for r in rs:
    xs.append(xs[-1]*0.9 + r)
df = pd.DataFrame(xs, columns=['data'])

# Find local peaks
df['min'] = df.data[(df.data.shift(1) > df.data) & (df.data.shift(-1) > df.data)]
df['max'] = df.data[(df.data.shift(1) < df.data) & (df.data.shift(-1) < df.data)]

# Plot results
plt.scatter(df.index, df['min'], c='r')
plt.scatter(df.index, df['max'], c='g')
df.data.plot()

edited Dec 29 '17 at 17:04

answered Dec 29 '17 at 14:33

fuglede

17,388
2
54
99

1

I found that when the values of data are repeated for example multiple rows with the value 7, using just < or > would miss the data point as a 'min' or a 'max'. Modifying this solution to have ".shift(1) <=" and ".shift(1) >=" did in fact allow for the identification of 'min' and 'max' values for repeated values. The logic is that the final row containing the repeated value will be treated as the 'min' or 'max'. – Udesh May 07 '20 at 03:40
great findins Udesh – sam May 11 '21 at 03:26
Great solution! – Saeed Apr 06 '22 at 23:34

score 3 · Answer 3 · answered Dec 27 '19 at 22:39

using Numpy

ser = np.random.randint(-40, 40, 100) # 100 points
peak = np.where(np.diff(ser) < 0)[0]

or

double_difference = np.diff(np.sign(np.diff(ser)))
peak = np.where(double_difference == -2)[0]

using Pandas

ser = pd.Series(np.random.randint(2, 5, 100))
peak_df = ser[(ser.shift(1) < ser) & (ser.shift(-1) < ser)]
peak = peak_df.index

score 3 · Answer 4 · edited Dec 30 '22 at 20:31

You can do something similar to Foad's .argrelextrema() solution, but with the Pandas .rolling() function:

# Find local peaks
n = 5 #rolling period
local_min_vals = df.loc[df['data'] == df['data'].rolling(n, center=True).min()]
local_max_vals = df.loc[df['data'] == df['data'].rolling(n, center=True).max()]

plt.scatter(local_min_vals.index, local_min_vals, c='r')
plt.scatter(local_max_vals.index, local_max_vals, c='g')

Pandas finding local max and min

4 Answers4

Linked

Related