MySQL / MariaDB: how to find gaps in timebased data?

Question

A logger system saves every 5 seconds a row of data (seconds==0,5,10,15,...,55; a time like 23:00:07 is not possible).

Sometimes the logger doesn't save because of a communication error, and rows are simply missing from the table.

I need to detect these gaps: I want to read the last line before the gap and the first line after the gap.

These are the demo data:

create table #time (
    DateTime datetime not null,
    Value int not null
);

insert into #time (DateTime, Value)
    values
        ('2018-08-23 00:00:00', 123),
        ('2018-08-23 00:00:05', 152),
        ('2018-08-23 00:00:10',  37),
        ('2018-08-23 00:00:15', 141),
        ('2018-08-23 00:00:20',  41),
        ('2018-08-23 00:00:25',  35),
        ('2018-08-23 00:00:30', 143),
        ('2018-08-23 00:00:35',  67),
        ('2018-08-23 00:00:40', 111),
                                        /* gap of one minute here */
        ('2018-08-23 00:01:45', 123),
        ('2018-08-23 00:01:50', 145),
        ('2018-08-23 00:01:55', 141),
        ('2018-08-23 00:02:00',  87),
                                        /* gap of 10 seconds here */
        ('2018-08-23 00:02:15', 190),
        ('2018-08-23 00:02:20', 122),
        ('2018-08-23 00:02:25', 123);

select * from #time;

They are also at RexTester

I would like to read back:

    GapStart              GapEnd                Size
0   2018-08-23 00:00:40   2018-08-23 00:01:45   0000-00-00 00:01:45
1   2018-08-23 00:02:00   2018-08-23 00:02:15   0000-00-00 00:00:15

Of course gaps of 5 seconds shouldn't be listed

What I found is referred to start/end DateTimes in the same row (Find gaps in timesheet data between certain hours), or is too complicated for me to be adapted to my case (find gaps in sequential dates).

I'm using MariaDB 10.0.28 (it is not possible to upgrade it): it means LAG() is not available.

Thank you in advance

You gave a SQL Server demo, so I ported your code over to a MySQL demo, and then updated the link. — Tim Biegeleisen, Aug 23 '18 at 10:26

score 3 · Accepted Answer · edited Aug 23 '18 at 11:14

One approach might be to first subquery and pair each record with the record having the nearest greater timestamp. Then, query that, and return all records having a gap of a sufficient size.

SELECT
    DateTime AS GapStart,
    NextDateTime AS GapEnd,
    TIMESTAMPDIFF(SECOND, DateTime, NextDateTime) AS SizeInSecond
FROM
(
    SELECT DateTime, Value,
        (SELECT MIN(DateTime) FROM #time t2
         WHERE t2.DateTime > t1.DateTime) AS NextDateTime
    FROM #time t1
) t
WHERE
    TIMESTAMPDIFF(SECOND, DateTime, NextDateTime) > 5;   -- or whatever threshhold you want

Demo

ADD ON to original answer

If DateTime is always growing, speed improvement can be obtained by changing the internal SELECT:

SELECT
    DateTime AS GapStart,
    NextDateTime AS GapEnd,
    TIMESTAMPDIFF(SECOND, DateTime, NextDateTime) AS SizeInSecond
FROM
(
    SELECT DateTime, Value,
        (SELECT DateTime FROM #time t2
         WHERE t2.DateTime > t1.DateTime LIMIT 1) AS NextDateTime
    FROM #time t1
) t
WHERE
    TIMESTAMPDIFF(SECOND, DateTime, NextDateTime) > 5;

MySQL / MariaDB: how to find gaps in timebased data?

1 Answers1

Demo

Linked