derivative of dataset in Pandas

Question

I have a cyclical dataset (pic and data below) that I'm trying to take the derivative of, dy/dt. I'd like the derivative for each cycle (rise), not just for each point like I am doing here. Part of the problem is the datetime of the samples are not evenly spaced. Ideally I'd like to get the slope of each cycle from minimum to maximum. This is an idealized case, my whole dataset would have noise and the slopes for each cycle would not necessarily be the same as this modeled set.

the data looks this,

time,y
12/15/18 01:10 AM,130352.146180556
12/16/18 01:45 AM,130355.219097222
12/17/18 01:47 AM,130358.223263889
12/18/18 02:15 AM,130361.281701389
12/19/18 03:15 AM,130364.406597222
12/20/18 03:25 AM,130352.427430556
12/21/18 03:27 AM,130355.431597222
12/22/18 05:18 AM,130358.663541667
12/23/18 06:44 AM,130361.842430556
12/24/18 07:19 AM,130364.915243056
12/25/18 07:33 AM,130352.944409722
12/26/18 07:50 AM,130355.979826389
12/27/18 09:13 AM,130359.153472222
12/28/18 11:53 AM,130362.4871875
12/29/18 01:23 PM,130365.673263889
12/30/18 02:17 PM,130353.785763889
12/31/18 02:23 PM,130356.798263889
01/01/19 04:41 PM,130360.085763889
01/02/19 05:01 PM,130363.128125

Here is my code,

import pandas as pd
import numpy as np
import matplotlib.pyplot as plot
from datetime import date, timedelta
import datetime

df=pd.read_csv('saw_data.csv') 
df['time']=pd.to_datetime(df['time'])

Set the date time to the index,

df.set_index(df['time'], inplace=True)

Here i am attempting to find the time interval between each data point.

df['Time_diff'] = pd.to_timedelta(df['time']-df['time'].shift()).dt.total_seconds().div(60)
#I believe units are minutes. For 'y' row 0 to 1, the diff is ~3 in about a day (86400 sec)
# so 3/86400 x 60 sec/min yields similar result to slope of 0.002 #/min.

'Time_diff' is from SO post: calculate the time difference between two consecutive rows in pandas

And this is suppose to be the derivative,

df['slope']=np.gradient(df['y'],1)/df['Time_diff']

Here are the results,

print(df.head())

                                   time              y  Time_diff     slope
time                                                                       
2018-12-15 01:10:00 2018-12-15 01:10:00  130352.146181        NaN       NaN
2018-12-16 01:45:00 2018-12-16 01:45:00  130355.219097     1475.0  0.002060
2018-12-17 01:47:00 2018-12-17 01:47:00  130358.223264     1442.0  0.002102
2018-12-18 02:15:00 2018-12-18 02:15:00  130361.281701     1468.0  0.002106
2018-12-19 03:15:00 2018-12-19 03:15:00  130364.406597     1500.0 -0.002951

Here is the dataset pic,

Are you essentially saying you want to get the gradient of each of the lines in the graph? — Hayden Eastwood, Jan 11 '20 at 17:06

score 0 · Answer 1 · answered Jan 11 '20 at 18:57

Ok, the below code assumes you're just using a classical "ruler and pen" method for getting the gradient (denoted as "m") - ie y distance / x distance, obtained from the difference in coordinates between each point.

The below code will give you the positive gradients (which I'm assuming is what you want). If you want the negative gradients (ie the lines slanting in the opposite direction), you will need to adjust the shift() method slightly.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plot
from datetime import date, timedelta
import datetime, time

df=pd.read_csv('saw_data.csv')
df['time']=pd.to_datetime(df['time'])
df.set_index(df['time'], inplace=True)

df['Time_diff'] = pd.to_timedelta(df['time']-df['time'].shift()).dt.total_seconds().div(60)
df['slope']=np.gradient(df['y'],1)/df['Time_diff']
#df['y_diff'] = pd.to_timedelta(df['y']-df['y'].shift()).dt.total_seconds().div(60)

df['time+1']=df['time'].shift()
df['y+1']=df['y'].shift()
df['y_diff'] = df['y'] - df['y+1']
df['m']= df['y_diff']/df['Time_diff']

You probably did a better job than me at finding the gradient, I got similar results. I was hoping to be able to find the gradient for each positive run, so (ytop-ybottom)/dt. — merit_2, Jan 11 '20 at 19:22
That's exactly what the result is, though. You're going to have to explain your problem more articulately to get more productive help. — Hayden Eastwood, Jan 11 '20 at 22:07

derivative of dataset in Pandas

1 Answers1