Split a series on time gaps in pandas?

Question

Is it possible to split a time series on it's gaps. For example, suppose we had the following:

rng2011 = pd.date_range('1/1/2011', periods=72, freq='H')
rng2012 = pd.date_range('1/1/2012', periods=72, freq='H')
Y = rng2011.union(rng2012)

Is it possible to look for gaps of a year or more, and split the data frame on them?

I imagine this would go something like:

Y.groupby(Y.map(lambda x: x.year))

Except that this splits on the year date, and I'm interested in specifying an interval gap rather than the year attribute of the row.

The application is I've got trip logs from a gps, but no delineation of when one trip ended and another began. I'd like to split on gaps of ten minutes or longer.

score 13 · Accepted Answer · edited Sep 03 '21 at 15:54

13

Assuming Y is a column in your dataframe, one way is to use diff and cumsum:

df = DataFrame(Y)
df[1] = df[0].diff() > 600000000000.0 #nanoseconds in ten minutes
df[1] = df[1].cumsum()
df.groupby(1)

Note: If you use the number of nanoseconds in 72 hours it'll split into two groups.

edited Sep 03 '21 at 15:54

Codec

3
4

answered Dec 20 '12 at 17:42

Andy Hayden

359,921
101
625
535

thanks, that works nicely! Can you also do the same with Y as an index? – Maus Dec 20 '12 at 20:04
does this work on 0.10.0? col 1 contains the range 0..143 on my system. – Garrett Dec 20 '12 at 20:17
@crewbum that's because the dataset is separated by hour (so they all split...)! – Andy Hayden Dec 20 '12 at 20:21
does it make sense for diff() on a datetime64 series to return a float64 series, while ``df[0][1] - df[0][0]`` returns a datetime.timedelta object? – Garrett Dec 20 '12 at 20:21
@Maus a lazy solution: `df[0] = df.index`... :s – Andy Hayden Dec 20 '12 at 20:22
@hayden definitely did that :) – Maus Dec 20 '12 at 21:43
3

diff returns timedelta objects now in pandas master. – K.-Michael Aye Mar 08 '13 at 05:54
... or, diff() will soon return timedelta: https://github.com/pydata/pandas/issues/3100 – Garrett Mar 29 '13 at 18:15
@K.-MichaelAye @crewburn Does it? I can still do this same thing in latest master... ('0.11.0.dev-184f2db'). `df = pd.DataFrame(pd.date_range('1/1/2011', periods=3, freq='H')); df[0].diff()[1] == 3600000000000.0`. – Andy Hayden Mar 29 '13 at 19:05
Yeah, sorry. But *NOW* it's working (fixed 7 days ago by the glorious Jeff). Just tried it with 0.12.0.dev-f354548. – K.-Michael Aye May 17 '13 at 01:00
@K.-MichaelAye Hurray! Good work Jeff (as always)! So it should land in 0.11.1 then :) – Andy Hayden May 17 '13 at 01:03
1

You could also use datetime.timedelta to express gap in larger units: df[1] = df[0].diff() > timedelta(minutes=10) – Yasir Jan Oct 13 '19 at 19:52

Split a series on time gaps in pandas?

1 Answers1

Linked