116

I have thousands of photos that were taken in Tanzania and I want to store the date and time each photo was taken in a MySQL database. The server, however, is located in the U.S. and I run into problems when I try to store a Tanzanian date-time that falls within the "invalid" hour during spring Daylight Savings time (in the U.S.). Tanzania doesn't do DST, so the time is an actually valid time.

Additional complications are that there are collaborators from many different timezones who will need to access the date-time values stored in the database. I want them to always come out as Tanzanian time and not in the local times that various collaborator are in.

I'm reluctant to set session times because I know that there will be problems when someone sometime forgets to set a session time and gets the times out all wrong. And I do not have authority to change anything about the server.

I've read: Daylight saving time and time zone best practices and MySQL datetime fields and daylight savings time -- how do I reference the "extra" hour? and Storing datetime as UTC in PHP/MySQL

But none of them seems to address my particular problem. I'm not an SQL expert; is there a way to specify timezone when setting DATETIMEs? I haven't seen one. Otherwise, any suggestions on how to approach this issue is greatly appreciated.

Edit: Here's an example of the problem I'm running into. I send the command:

INSERT INTO Images (CaptureEvent, SequenceNum, PathFilename, TimestampJPG) 
VALUES (122,1,"S2/B04/B04_R1/IMAG0148.JPG","2011-03-13 02:49:10")

And I get the error:

Error 1292: Incorrect datetime value: '2011-03-13 02:49:10' for column 'TimestampJPG'

This date and time exists in Tanzania, but not in the U.S., where the database is.

Community
  • 1
  • 1
mkosmala
  • 1,437
  • 3
  • 12
  • 10
  • 24
    You shouldn't want to store time zone info in database. Store all date/time data as UTC and always make time zone offset adjustment on the application layer. – marekful Nov 07 '13 at 18:00
  • 8
    @MarcellFülöp: I keep seeing people say "store it as UTC" but I don't understand what that means. How can I store a date time "as UTC"? As far as I can tell all I can do is insert something of the form YYYY-MM-DD HH:MM:SS. Where do I tell the database that it's UTC? – mkosmala Nov 07 '13 at 18:03
  • 1
    You don't. MySQL by default will use the system time zone internally but it's possible to define a different time zone for the MySQL server globally or even per transaction. When you insert a date, it is not possible to define the time zone along with the date string in MySQL. A date like '2013-11-10 00:00' refers to a point in time since the epoch. You store it and you know what time zone your server is in. Then when you retrieve it, you can make the necessary adjustments to transform that date form the server's time zone to the client's. – marekful Nov 07 '13 at 18:12
  • 1
    @marekful Nope, it depends. I want to filter out photos that were taken at dawn, and UTC won't help in this case. I would like to save both the UTC timestamp and the timezone offset. – Arnie97 Jul 08 '21 at 16:38
  • 2
    The best practice, IMO and what many experienced engineers follow, is still to store all date/time values as UTC (a.k.a. Zulu). Configure the db server to be in UTC regardless of physical location. Then, what you want to do in this particular case is to know that you want to find times of day that correspond to dawn _in Tanzania_. So you add the logic in your program code. Create a time zone instance with Tanzania's time zone and filter all date/time values fetched from the database through it. It will do the rest and represent all values in Tanzania time. It will also take care of DST. – marekful Jul 12 '21 at 16:26
  • @marekful if I store date/time as UTC how to do I know what is the time zone from that column or I need another column to store time zone? – IGRACH Aug 26 '21 at 11:20
  • UTC does not have time zones. You convert any local time to UTC (i.e. adjust by offset and possible DST) before storing it in the database. You also apply local time zone / DST to the value where you display it. E.g. it's 22:00 in a location with UTC+2 then you store 20:00 and when you need to display it to someone in a UTC-4 location then you adjust the stored 20:00 to get 16:00. – marekful Aug 26 '21 at 23:17
  • @marekful Sounds all good in theory, until you want to respect DST in your stored times. The client shifts the clock and so should the time if its record is not changed. UTC date format will prevent time from moving with DST events. – Martin Braun Feb 22 '23 at 06:04
  • @MartinBraun DST shouldn't be a problem the same way time zones shouldn't because when you store a date/time value as a Unix timestamp, that value represents a fixed number of seconds relative relative to the Unix Epoch, all in UTC. (UTC is independent of time zones and DST.) When this timestamp is retrieved in the future and needs to be displayed on a client's screen, then you know that the universal time at the epoch + the given number of seconds gives you the target date/time in UTC. You also know what time zone offset the client operates in, so you adjust the target date for that. – marekful Feb 28 '23 at 00:54
  • You also know whether DST was observed at that time and you also know whether DST is observed at the future "now" so these are used to adjust the target date for DST. The point is that this target date IS based on the the original UTC timestamp and internally mutated to account for regional differences. – marekful Feb 28 '23 at 00:54
  • @marekful That was my point, you can't do that if you want the stored time to be shifted as well. For fixed times your approach is right, storing UTC or epoch is common sense. It's not recommend when you store a time that needs to be dependent on DST, for instance: The user wants to store the open and closing times of a restaurant. In this case, you should store the time without any timezone information, so that it will be consistent on the clock face at their location. In this case it's not about storing a moment in time, I felt the need to address this edge-case. – Martin Braun Feb 28 '23 at 16:27

8 Answers8

91

You said:

I want them to always come out as Tanzanian time and not in the local times that various collaborator are in.

If this is the case, then you should not use UTC. All you need to do is to use a DATETIME type in MySQL instead of a TIMESTAMP type.

From the MySQL documentation:

MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME.)

If you are already using a DATETIME type, then you must be not setting it by the local time to begin with. You'll need to focus less on the database, and more on your application code - which you didn't show here. The problem, and the solution, will vary drastically depending on language, so be sure to tag the question with the appropriate language of your application code.

Matt Johnson-Pint
  • 230,703
  • 74
  • 448
  • 575
  • 2
    I am using a DATETIME. But the database throws an error if I try to set a time that doesn't exist in the U.S. (but does in Tanzania). I've edited my question to give an example of the trouble I run into. My application code happens to be in Python, but it's really just a wrapper around SQL code right now. I'm happy to do transformations in Python, but not sure what they should be. – mkosmala Nov 08 '13 at 17:52
  • 3
    Could you please show your python code then, so we can see how you are building the SQL query? You might just be dealing with naive vs aware datetimes in python. And are you *positive* that your `TimestampJPG` column is a `DATETIME` type in the mysql database schema and not a `TIMESTAMP` type? – Matt Johnson-Pint Nov 09 '13 at 05:22
  • 1
    Arg. You're right. I changed all the other TIMESTAMPs in the database to DATETIME, but somehow missed this one. That solves the issue. Thanks. – mkosmala Mar 31 '14 at 16:09
  • 9
    @poige Why do you say Postgresql will be better here? It also converts timestamp to UTC before storing. – Anshul Aug 21 '15 at 05:18
  • 1
    MariaDB devs are working on implementing DATETIME with timezones: https://jira.mariadb.org/browse/MDEV-11829 – mae Jun 14 '21 at 03:52
  • If we receive a legacy database in which there are not info about the timezone, How can we show the real date to the users? I think it is necessary to save the timezone too. – JRichardsz Jun 16 '22 at 00:02
59

None of the answers here quite hit the nail on the head.

How to store a datetime in MySQL with timezone info

Use two columns: DATETIME, and a VARCHAR to hold the time zone information, which may be in several forms:

A timezone or location such as America/New_York is the highest data fidelity.

A timezone abbreviation such as PST is the next highest fidelity.

A time offset such as -2:00 is the smallest amount of data in this regard.

Some key points:

  • Avoid TIMESTAMP because it's limited to the year 2038, and MySQL relates it to the server timezone, which is probably undesired.
  • A time offset should not be stored naively in an INT field, because there are half-hour and quarter-hour offsets.

If it's important for your use case to have MySQL compare or sort these dates chronologically, DATETIME has a problem:

'2009-11-10 11:00:00 -0500' is before '2009-11-10 10:00:00 -0700' in terms of "instant in time", but they would sort the other way when inserted into a DATETIME.

You can do your own conversion to UTC. In the above example, you would then have
'2009-11-10 16:00:00' and '2009-11-10 17:00:00' respectively, which would sort correctly. When retrieving the data, you would then use the timezone info to revert it to its original form.

One recommendation which I quite like is to have three columns:

  • local_time DATETIME
  • utc_time DATETIME
  • time_zone VARCHAR(X) where X is appropriate for what kind of data you're storing there. (I would choose 64 characters for timezone/location.)

An advantage to the 3-column approach is that it's explicit: with a single DATETIME column, you can't tell at a glance if it's been converted to UTC before insertion.


Regarding the descent of accuracy through timezone/abbreviation/offset:

  • If you have the user's timezone/location such as America/Juneau, you can know accurately what the wall clock time is for them at any point in the past or future (barring changes to the way Daylight Savings is handled in that location). The start/end points of DST, and whether it's used at all, are dependent upon location, so this is the only reliable way.
  • If you have a timezone abbreviation such as MST, (Mountain Standard Time) or a plain offset such as -0700, you will be unable to predict a wall clock time in the past or future. For example, in the United States, Colorado and Arizona both use MST, but Arizona doesn't observe DST. So if the user uploads his cat photo at 14:00 -0700 during the winter months, was he in Arizona or California? If you added six months exactly to that date, would it be 14:00 or 13:00 for the user?

These things are important to consider when your application has time, dates, or scheduling as core function.


References:

Matt Mc
  • 8,882
  • 6
  • 53
  • 89
  • 4
    I'm not seeing benefit to storing local time. IMHO, the cleanest way to store the datetime value itself, is in UTC. This avoids any confusion when comparing datetimes. If there is a reason to know what that was in a particular local time, then record a LOCATION. App logic can then apply that location to know local time (e.g. was it during work hours?) BUT keep the DateTime itself in UTC. E.g. our Swedish facility is defined to be in Sweden's timezone; we already have a field (in an event record) saying "at what facility", so don't need to SEPARATELY add a timezone to every event record. – ToolmakerSteve Mar 31 '20 at 17:45
  • 4
    The benefit of keeping dates in UTC becomes apparent if someone at one location is entering an event for a different location. If technician in New York reports an event in Sweden, is the "local time" New York, or Sweden? Avoid this confusion by storing a UTC time. The technician's location and the facility's location, attached to the event, allow it to be viewed in *either* "local time", as desired, without any confusion - simply by specifying POV. Or you might be interested in when it happened in YOUR time, which might be a third "local time". Specify "local offset" dynamically via location. – ToolmakerSteve Mar 31 '20 at 18:00
  • 7
    @ToolmakerSteve If you read my answer again, you'll see that my primary point is that you need to store `time` and `location` in separate columns. I also talk about storing the UTC time. If you prefer to store *only* the UTC time because your `location`s are plural and/or in another table, great. My only suggestion then is to label it `utc_time` for clarity. – Matt Mc Mar 31 '20 at 18:41
  • If I need time for any datapoints that are not database's metadata, I also use this approach. For example I store some travel data that goes across multiple timezones. I keep four columns (start_dt, start_tz, end_dt, end_tz), but for simplicity I also calculate `duration` so that I don't need to do those calculations all the time. If I need to display them in a users's timezone, I do it in the application layer.. Though sometimes I am thinking of changing to storing UTC + timezone, so that I can pass UTC to application more easily – Moseleyi Jan 05 '21 at 21:54
  • 3
    Another reason to store `utc_time` + `tz` instead of `local_time` + `tz` is that `local_time` + `tz` doesn't always unambiguously could be converted to point in time (an instant). E.g. when DST time clocks if shifted 1 hour back at 3:00 am, there will be two instances of 2:01 AM local time and they will be indistinguishable from each other, even if you know the TZ id. So, to store both: "the moment in time" and "what time it was on the local clocks at the moment", you need to store utc time + tz id. Or local time + offset + tz id (sometimes it is preferable). – Ruslan Stelmachenko Dec 28 '21 at 16:37
13

MySQL stores DATETIME without timezone information. Let's say you store '2019-01-01 20:00:00' into a DATETIME field, when you retrieve that value you're expected to know what timezone it belongs to.

So in your case, when you store a value into a DATETIME field, make sure it is Tanzania time. Then when you get it out, it will be Tanzania time. Yay!

Now, the hairy question is: When I do an INSERT/UPDATE, how do I make sure the value is Tanzania time? Two cases:

  1. You do INSERT INTO table (dateCreated) VALUES (CURRENT_TIMESTAMP or NOW()).

  2. You do INSERT INTO table (dateCreated) VALUES (?), and specify the current time from your application code.

CASE #1

MySQL will take the current time, let's say that is '2019-01-01 20:00:00' Tanzania time. Then MySQL will convert it to UTC, which comes out to '2019-01-01 17:00:00', and store that value into the field.

So how do you get the Tanzania time, which is '20:00:00', to store into the field? It's not possible. Your code will need to expect UTC time when reading from this field.

CASE #2

It depends on what type of value you pass as ?. If you pass the string '2019-01-01 20:00:00', then good for you, that's exactly what will be stored to the DB. If you pass a Date object of some kind, then it'll depend on how the db driver interprets that Date object, and ultimate what 'YYYY-MM-DD HH:mm:ss' string it provides to MySQL for storage. The db driver's documentation should tell you.

Sarsaparilla
  • 6,300
  • 1
  • 32
  • 21
  • 5
    Case #1 is wrong. MySQL does not convert values for datetime columns to UTC. They are stored as is. – ferow2k Mar 19 '21 at 22:17
  • Actually it depends on the TZ of the server. A lot of people put servers on cloud in UTC, in that case the answer is right. – capr Mar 17 '23 at 15:47
  • When I reread what I wrote, I realize it could be confusing. Perhaps the main thing to keep in mind is MySQL only stores the string 'YYYY-MM-DD HH:mm:ss' for a DATETIME field. Whatever string you give to MySQL is what it stores. If you give an object that isn't a string, then you need to figure out what datetime string that object gets converted to. – Sarsaparilla Mar 17 '23 at 23:48
7

All the symptoms you describe suggest that you never tell MySQL what time zone to use so it defaults to system's zone. Think about it: if all it has is '2011-03-13 02:49:10', how can it guess that it's a local Tanzanian date?

As far as I know, MySQL doesn't provide any syntax to specify time zone information in dates. You have to change it a per-connection basis; something like:

SET time_zone = 'EAT';

If this doesn't work (to use named zones you need that the server has been configured to do so and it's often not the case) you can use UTC offsets because Tanzania does not observe daylight saving time at the time of writing but of course it isn't the best option:

SET time_zone = '+03:00';
Álvaro González
  • 142,137
  • 41
  • 261
  • 360
  • 17
    You can not just slap on a UTC offset though because it ignores daylight savings time. Your time could easily end up an hour off with little hope of ever correcting it! I know of functions in SQL Server that will convert UTC to your timezone but it does not take into account any arbitrary shift in daylight savings time. Store in UTC zero offset and convert at the application--this solves a lot of problems very cleanly but you will have a hell of a time converting dates properly in ad-hoc queries. – Sean Anderson Nov 15 '14 at 04:12
  • Now it is possible to specify at least the time zone offset directly in the date e.g. like this: '2020-01-01 10:10:10+05:30' (from https://dev.mysql.com/doc/refman/8.0/en/datetime.html ). Because the time zone offset is for a specific date and time it does not matter that it does not specify a specific time zone (and thus their daylight saving time rules). Because this way you exactly specify how UTC can be computed from that date and MySQL saves it as UTC in its database. A time zone like "EAT" is only necessary if you want to compute UTC for multiple dates (DST and not DST). – Dominique Nov 10 '20 at 08:24
5

Beginning with MySQL 8.0.19, you can specify a time zone offset when inserting TIMESTAMP and DATETIME values into a table.

The offset is appended to the time part of a datetime literal, with no intervening spaces, and uses the same format used for setting the time_zone system variable, with the following exceptions:

  • For hour values less than 10, a leading zero is required.
  • The value '-00:00' is rejected.
  • Time zone names such as 'EET' and 'Asia/Shanghai' cannot be used; 'SYSTEM' also cannot be used in this context.

The value inserted must not have a zero for the month part, the day part, or both parts. This is enforced beginning with MySQL 8.0.22, regardless of the server SQL mode setting.


EXAMPLE:

This example illustrates inserting datetime values with time zone offsets into TIMESTAMP and DATETIME columns using different time_zone settings, and then retrieving them:

mysql> CREATE TABLE ts (
    ->     id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
    ->     col TIMESTAMP NOT NULL
    -> ) AUTO_INCREMENT = 1;

mysql> CREATE TABLE dt (
    ->     id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    ->     col DATETIME NOT NULL
    -> ) AUTO_INCREMENT = 1;

mysql> SET @@time_zone = 'SYSTEM';

mysql> INSERT INTO ts (col) VALUES ('2020-01-01 10:10:10'),
    ->     ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');

mysql> SET @@time_zone = '+00:00';

mysql> INSERT INTO ts (col) VALUES ('2020-01-01 10:10:10'),
    ->     ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');

mysql> SET @@time_zone = 'SYSTEM';

mysql> INSERT INTO dt (col) VALUES ('2020-01-01 10:10:10'),
    ->     ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');

mysql> SET @@time_zone = '+00:00';

mysql> INSERT INTO dt (col) VALUES ('2020-01-01 10:10:10'),
    ->     ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');

mysql> SET @@time_zone = 'SYSTEM';

mysql> SELECT @@system_time_zone;
+--------------------+
| @@system_time_zone |
+--------------------+
| EST                |
+--------------------+

mysql> SELECT col, UNIX_TIMESTAMP(col) FROM dt ORDER BY id;
+---------------------+---------------------+
| col                 | UNIX_TIMESTAMP(col) |
+---------------------+---------------------+
| 2020-01-01 10:10:10 |          1577891410 |
| 2019-12-31 23:40:10 |          1577853610 |
| 2020-01-01 13:10:10 |          1577902210 |
| 2020-01-01 10:10:10 |          1577891410 |
| 2020-01-01 04:40:10 |          1577871610 |
| 2020-01-01 18:10:10 |          1577920210 |
+---------------------+---------------------+

mysql> SELECT col, UNIX_TIMESTAMP(col) FROM ts ORDER BY id;
+---------------------+---------------------+
| col                 | UNIX_TIMESTAMP(col) |
+---------------------+---------------------+
| 2020-01-01 10:10:10 |          1577891410 |
| 2019-12-31 23:40:10 |          1577853610 |
| 2020-01-01 13:10:10 |          1577902210 |
| 2020-01-01 05:10:10 |          1577873410 |
| 2019-12-31 23:40:10 |          1577853610 |
| 2020-01-01 13:10:10 |          1577902210 |
+---------------------+---------------------+

Note:

  • Sadly, the offset is not displayed when selecting a datetime value, even if one was used when inserting it.
  • The range of supported offset values is -13:59 to +14:00, inclusive.
  • Datetime literals that include time zone offsets are accepted as parameter values by prepared statements.
  • MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME.)
  • This is a great solution. I prefer this approach as I can simply specify the timezone as part of the value when inserting a time. This way I don't have to worry about implicit timezones of client, server or connection. I wish MariaDB would support it too... – jgivoni Jul 25 '23 at 09:30
2

You can't... you will find a lot of answers saying you "it is not necessary, store using UTC", but it is: you need to store datetimes with the timezone and MySQL can't...

I worked last 10 years in Postgres and all this kind of problems doesn't exist (date times and timezones are managed with no friction, you can store and compare datetimes expressed in different time zones transparently, the ISOString format is managed naturally,etc...).

I actually work in MariaDB and I can't understand why in 2022, in a globalized world, MySQL continues not supporting per value timezones.

ddcovery
  • 157
  • 1
  • 6
0

I once also faced such an issue where i needed to save data which was used by different collaborators and i ended up storing the time in unix timestamp form which represents the number of seconds since january 1970 which is an integer format. Example todays date and time in tanzania is Friday, September 13, 2019 9:44:01 PM which when store in unix timestamp would be 1568400241

Now when reading the data simply use something like php or any other language and extract the date from the unix timestamp. An example with php will be

echo date('m/d/Y', 1568400241);

This makes it easier even to store data with other collaborators in different locations. They can simply convert the date to unix timestamp with their own gmt offset and store it in a integer format and when outputting this simply convert with a

Geoff
  • 6,277
  • 23
  • 87
  • 197
  • 3
    Technically speaking you aren't storing dates with time zone info—you're *discarding* the time zone info. That's basically what native date types already do in MySQL ;-) – Álvaro González Sep 14 '19 at 07:24
0

for me the simplest solution was storing the time zone offset as an int. To allow for half and quarter hours zones I store time offset in minutes instead of hours. For example: UTC-06 would be -360

  • This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://stackoverflow.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://stackoverflow.com/help/whats-reputation), you can also [add a bounty](https://stackoverflow.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/34441089) – treckstar May 26 '23 at 06:28