Converting time string into epoch time on windows

Question

I have an interface where I'm getting a file and the file name is the time stamp when the internal data was valid. I'm processing the files in order and providing them to another API which takes the time as milliseconds since Jan 1st, 1970.

So, I get the file name in as a string in the format of YYYYMMDD_HHmmSS.sss and have to turn this into time. I assume the processing and collection occurred in the same timezone and such and simply have to convert the characters to digits.

uint64_t foo::convertFileNameToTimestamp(const std::string& filename) const
{
    const uint64_t yyyy = atoi(filename.substr(0,4).c_str());
    const uint64_t MM = atoi(filename.substr(4,2).c_str());
    const uint64_t DD = atoi(filename.substr(6,2).c_str());
    const uint64_t HH = atoi(filename.substr(9,2).c_str());
    const uint64_t MI = atoi(filename.substr(11,2).c_str());
    const uint64_t SS = atoi(filename.substr(13,2).c_str());
    const uint64_t sss = atoi(filename.substr(16,3).c_str());

    // number of milliseconds in a day
    const uint64_t DAY = 24 * 60 * 60 * 1000;
    int MD = 0;
    if (MM == 2) MD = 31; // currently feb, so add all of january's days
    else if (MM == 3) MD = 31+28; // ...
    else if (MM == 4) MD = 31+28+31;
    else if (MM == 5) MD = 31+28+31+30;
    else if (MM == 6) MD = 31+28+31+30+31;
    else if (MM == 7) MD = 31+28+31+30+31+30;
    else if (MM == 8) MD = 31+28+31+30+31+30+31;
    else if (MM == 9) MD = 31+28+31+30+31+30+31+31;
    else if (MM == 10) MD = 31+28+31+30+31+30+31+31+30;
    else if (MM == 11) MD = 31+28+31+30+31+30+31+31+30+31;
    else if (MM == 12) MD = 31+28+31+30+31+30+31+31+30+31+30;

    // year 2000 wasn't a leap year
    uint64_t YLD = ((yyyy-1970) / 4);
    if (yyyy > 2000) --YLD;

    uint64_t temp = sss;
    temp += SS * 1000;
    temp += MI * 60 * 1000;
    temp += HH * 60 * 60 * 1000;
    temp += (DD-1) * DAY;
    temp += (MD) * DAY;
    temp += (yyyy-1970) * 365 * DAY + YLD*DAY;

    return temp;
}

Obviously, reinventing the wheel here. Seems like there should be some sort of function for this. Also.. how do I account for leap seconds? It was annoying enough to deal with leap days. The time stamps are all from 2015 and beyond and always will be, but I don't think I can just blindly add 26 seconds. Eventually we will have 27 or back up to 25. In previous functions I have validated the string to be the right format and the file to exist and all that jazz. I'm running on windows 8.1 compilation for 64 bit using VS 2010.

I've looked at Convert Epoch Time string to Time which suggested ctime(), but it doesn't seem to deal with milliseconds in the constructor or even any of the get methods and it doesn't accept generically formatted string inputs. I'm assuming I've got to call some time classes CTOR which will accept the file name string and then call some accessor on it to get the millisecond time since 1970 including leap seconds and such.

I am not using boost and don't have access/permission to use it.

That `else if` block is atrocious. Put the days into a datastructure and loop back over them using `MM` as the start index — user1231232141214124, Aug 25 '15 at 18:25
Date algorithms you may find useful: http://howardhinnant.github.io/date_algorithms.html — Howard Hinnant, Aug 26 '15 at 03:36
@HowardHinnant quite right. Found that out when I checked against the answer below. — LawfulEvil, Aug 26 '15 at 12:23

CristiFati · Answer 1 · 2015-08-26T18:25:16.983

You can use this piece of code (you don't have to worry about leap years and all the related stuff). @Edit1: Modified the code to take leap seconds into account; also restructured it into a class:

Foo.h:

#ifndef __FOO__H__
#define __FOO__H__

#include <string>
#include <Windows.h>
#include <stdint.h>

class CFoo {
private:
    const static int kLeapSecsDim = 26;
    static uint64_t msecsBetweenEpochs;
    static SYSTEMTIME leapSecs[kLeapSecsDim];

    ULARGE_INTEGER leapSecsUi[kLeapSecsDim];

    int CFoo::getLeapSeconds(ULARGE_INTEGER ui) const;
public:
    CFoo();
    ~CFoo() {};

    uint64_t toEpoch(const std::string& filename) const;
};

#endif  //__FOO__H__

Foo.cpp:

#include "Foo.h"

uint64_t CFoo::msecsBetweenEpochs = 11644473600000; /* Milliseconds between 1.1.1601 and 1.1.1970 */
SYSTEMTIME CFoo::leapSecs[CFoo::kLeapSecsDim] = 
                            {{1972, 06, 0, 30, 23, 59, 59, 999},
                             {1972, 12, 0, 31, 23, 59, 59, 999},
                             {1973, 12, 0, 31, 23, 59, 59, 999},
                             {1974, 12, 0, 31, 23, 59, 59, 999},
                             {1975, 12, 0, 31, 23, 59, 59, 999},
                             {1976, 12, 0, 31, 23, 59, 59, 999},
                             {1977, 12, 0, 31, 23, 59, 59, 999},
                             {1978, 12, 0, 31, 23, 59, 59, 999},
                             {1979, 12, 0, 31, 23, 59, 59, 999},
                             {1981, 06, 0, 30, 23, 59, 59, 999},
                             {1982, 06, 0, 30, 23, 59, 59, 999},
                             {1983, 06, 0, 30, 23, 59, 59, 999},
                             {1985, 06, 0, 30, 23, 59, 59, 999},
                             {1987, 12, 0, 31, 23, 59, 59, 999},
                             {1989, 12, 0, 31, 23, 59, 59, 999},
                             {1990, 12, 0, 31, 23, 59, 59, 999},
                             {1992, 06, 0, 30, 23, 59, 59, 999},
                             {1993, 06, 0, 30, 23, 59, 59, 999},
                             {1994, 06, 0, 30, 23, 59, 59, 999},
                             {1995, 12, 0, 31, 23, 59, 59, 999},
                             {1997, 06, 0, 30, 23, 59, 59, 999},
                             {1998, 12, 0, 31, 23, 59, 59, 999},
                             {2005, 12, 0, 31, 23, 59, 59, 999},
                             {2008, 12, 0, 31, 23, 59, 59, 999},
                             {2012, 06, 0, 30, 23, 59, 59, 999},
                             {2015, 06, 0, 30, 23, 59, 59, 999},
                            };


int CFoo::getLeapSeconds(ULARGE_INTEGER ui) const {
    int ret = 0;
    for (int i = 0; i < kLeapSecsDim; i++) {
        if (ui.QuadPart <= this->leapSecsUi[i].QuadPart)
            break;
        ret++;
    }
    return ret;
}

CFoo::CFoo() {
    FILETIME ft;
    BOOL res;
    for (int i = 0; i < this->kLeapSecsDim; i++) {
        res = SystemTimeToFileTime(&(this->leapSecs[i]), &ft);
        if (res == FALSE)
            throw std::exception("SystemTimeToFileTime error", GetLastError());
        this->leapSecsUi[i].LowPart = ft.dwLowDateTime;
        this->leapSecsUi[i].HighPart = ft.dwHighDateTime;
    }
}

uint64_t CFoo::toEpoch(const std::string& filename) const {
    SYSTEMTIME st;
    FILETIME ft;
    ULARGE_INTEGER ui;
    st.wYear = atoi(filename.substr(0, 4).c_str());
    st.wMonth = atoi(filename.substr(4, 2).c_str());
    st.wDay = atoi(filename.substr(6, 2).c_str());
    st.wHour = atoi(filename.substr(9, 2).c_str());
    st.wMinute = atoi(filename.substr(11, 2).c_str());
    st.wSecond = atoi(filename.substr(13, 2).c_str());
    st.wMilliseconds = atoi(filename.substr(16, 3).c_str());
    BOOL result = SystemTimeToFileTime(&st, &ft);
    if (result == FALSE)
        throw std::exception("SystemTimeToFileTime error", GetLastError());
    ui.HighPart = ft.dwHighDateTime;
    ui.LowPart = ft.dwLowDateTime;
    //printf("%016I64X - %I64u\n", ui.QuadPart, ui.QuadPart);
    //printf("%016I64X - %I64u\n", ui.QuadPart/10000, ui.QuadPart/10000);
    return (ui.QuadPart / 10000) - this->msecsBetweenEpochs + this->getLeapSeconds(ui) * 1000;
}

Notes:

For invalid dates/times SystemTimeToFileTime will fail
The constant CFoo::msecsBetweenEpochs I think it can be found on Google; I took it from Python(2.7.10)'s posixmodule.c (actually there's the seconds number; I only had to multiply it by 1000)
Your implementation yields results that are not very accurate (I used http://www.epochconverter.com for reference).
According to SystemTimeToFileTime, the timestamp is in UTC.

I will try it out and see what I get. Why divide by 10000? Is the ui.QuadPart in tens of nanoseconds? No worries on the wrapping, I'll make it fit into my architecture. — LawfulEvil, Aug 25 '15 at 20:11
Yes (almost :) ), here's [FILETIME MSDN page](https://msdn.microsoft.com/en-us/library/windows/desktop/ms724284(v=vs.85).aspx). — CristiFati, Aug 25 '15 at 20:22
So, the problem remains. This answer produces the same answer(once I dealt with year 2k having a leap day) as my solution, that is, it doesn't include leap seconds. It assumes the time string did not have leapseconds in it and doesn't add any itself. As this answer shows, the filetime/system time from windows doesn't deal with leapseconds. http://stackoverflow.com/questions/130573/does-the-windows-filetime-structure-include-leap-seconds — LawfulEvil, Aug 26 '15 at 13:07
Yes you're right! I don't know how why i thought that it would handle leap seconds (maybe i assumed so since it handles leap years). I can correct the solution by adding the leap seconds, in form of a table(data from http://www.timeanddate.com/time/leap-seconds-future.html), the drawback is that every time a new leap second is added, the code will need to be adjusted. Should i go on and adjust solution or should i simply delete it? — CristiFati, Aug 26 '15 at 16:47
Thanks. However, this is the sort of thing I wanted to avoid. If my code it maintaining the lookup table, then I need to update my code and re-release every time a leap second gets added. My hope was that windows would have the number of leap seconds included in some built in function and that when new leap seconds were added, it would be included in one of the weekly windows update packages. — LawfulEvil, Aug 26 '15 at 18:45
Yes, it makes sense. I imagined you're looking for a _Windows_ equivalent for `tzdata`.A quick search didn't reveal anything that would quickly solve the problem. Hmm, a workaround would be to store the table in a config file that gets read by the application; that way you'll only have to redistribute the config file, without recompiling. Or a nicer one would be to extend the app to get the leap second data from the internet, but that would require quite some changes. — CristiFati, Aug 26 '15 at 19:11
Here is a free, open source, cross-platform IANA tzdata parser for C++11/C++14: http://howardhinnant.github.io/tz.html You still have to keep your IANA database up to date. And it is a bit heavy-handed just for this application. Nevertheless, one could use it to add leap seconds to this application. Note though that it may actually be incorrect to add leap seconds in. Write a test where you know the right answer (and leap seconds will make a difference). — Howard Hinnant, Aug 28 '15 at 18:00
Thanks, I've downloaded myself a bunch of stuff (some from _IANA_), also the _tzdata_ sources, but seems that seems that on _Windows_ there's no way of doing this automatically. Found [this](https://www.microsoft.com/en-us/store/apps/tzdata/9nblgggzkfqg) that seem to be related but it's most likely useless in our case. Why would it be incorrect to add the leap seconds to the date? The _right answer_ is somehow hard to find :d — CristiFati, Aug 28 '15 at 20:48

Howard Hinnant · Accepted Answer · 2016-05-21T20:41:07.817

Here is an answer that will work on any platform that supports C++11 or C++14. It builds off of the std::chrono library that was introduced in C++11. It also uses a free, open source, cross platform library to simplify the arithmetic (MIT license which is usually considered lawyer-friendly).

If you don't need to take leap seconds into account, you can use this date library, and it will look like this:

#include <string>
#include "date.h"

using time_stamp = std::chrono::time_point<std::chrono::system_clock,
                                           std::chrono::milliseconds>;

time_stamp
convertFileNameToTimestamp(const std::string& filename)
{
    using namespace std::chrono;
    using namespace date;
    const uint64_t yyyy = atoi(filename.substr(0,4).c_str());
    const uint64_t MM = atoi(filename.substr(4,2).c_str());
    const uint64_t DD = atoi(filename.substr(6,2).c_str());
    const uint64_t HH = atoi(filename.substr(9,2).c_str());
    const uint64_t MI = atoi(filename.substr(11,2).c_str());
    const uint64_t SS = atoi(filename.substr(13,2).c_str());
    const uint64_t sss = atoi(filename.substr(16,3).c_str());
    return sys_days{year(yyyy)/MM/DD}
         + hours{HH} + minutes{MI} + seconds{SS} + milliseconds{sss};
}

After parsing the numbers out of the filename, it is very simple to create a type-safe std::chrono::time_point which simply holds an integral number of milliseconds since 1970-01-01 (as an int64_t).

If you want to take leap seconds into account, you need this higher-level library which is a complete parser of the IANA timezone database. You will also need to keep an updated copy of the IANA timezone database downloaded for my timezone/leap-second library to parse. But once set up, the source code for your converter is very similar to that above, and nearly as simple:

#include <string>
#include "tz.h"

using time_stamp_ls = std::chrono::time_point<date::utc_clock,
                                              std::chrono::milliseconds>;

time_stamp_ls
convertFileNameToTimestamp_ls(const std::string& filename)
{
    using namespace std::chrono;
    using namespace date;
    const uint64_t yyyy = atoi(filename.substr(0,4).c_str());
    const uint64_t MM = atoi(filename.substr(4,2).c_str());
    const uint64_t DD = atoi(filename.substr(6,2).c_str());
    const uint64_t HH = atoi(filename.substr(9,2).c_str());
    const uint64_t MI = atoi(filename.substr(11,2).c_str());
    const uint64_t SS = atoi(filename.substr(13,2).c_str());
    const uint64_t sss = atoi(filename.substr(16,3).c_str());
    return utc_clock::sys_to_utc(sys_days{year(yyyy)/MM/DD}
         + hours{HH} + minutes{MI} + seconds{SS} + milliseconds{sss});
}

Both of these functions can be exercised with the following HelloWorld:

#include <iostream>

int
main()
{
    std::string filename = "20150830_002120.123";
    std::cout << convertFileNameToTimestamp   (filename).time_since_epoch().count() << '\n';
    std::cout << convertFileNameToTimestamp_ls(filename).time_since_epoch().count() << '\n';
}

which outputs:

1440894080123
1440894106123

Note that these timestamps are exactly 26,000ms apart.

Update

The "tz.h" header now includes a parse function which makes writing these functions much easier:

date::sys_time<std::chrono::milliseconds>
convertFileNameToTimestamp(const std::string& filename)
{
    using namespace std::chrono;
    using namespace date;
    std::istringstream in{filename};
    sys_time<milliseconds> tp;
    parse(in, "%Y%m%d_%H%M%S", tp);
    return tp;
}

date::utc_time<std::chrono::milliseconds>
convertFileNameToTimestamp_ls(const std::string& filename)
{
    return date::to_utc_time(convertFileNameToTimestamp(filename));
}

Sorry for asking, is that the real answer to the question? , Does this get updated automatically on _Windows_? (the _stl_ lib gets updated every time a leap second comes out...)? — CristiFati, Sep 02 '15 at 02:16
@CristiFati: Sorry for the miscommunication. Except for the `` lib, which *does not* conveniently do calendars, the above libraries are *not* standard. They are things I've written and am offering as free and open source. On the issue of leap seconds, the higher-level tz library depends on *you* downloading an up-to-date copy of the IANA timezone database. This is all clearly spelled out in the documentation linked to in the answer. There *is no* good std::lib solution to this problem at this time (imho). — Howard Hinnant, Sep 02 '15 at 02:49

Converting time string into epoch time on windows

2 Answers2

Linked