1

I have two applications using the same postgres DB. A laravel application and a rails application. The current data in the DB is in Y-m-d H:i:s format but whenever rails adds a records, the format is Y-m-d H:i:s.u which includes milliseconds.

This causes the following error on the laravel side if the created_at date is ever references in laravel InvalidArgumentException Trailing data...

Laravel models can mutate their date format, so I can include make it conform to Y-m-d H:i:s.u then I would need to update all the records to have the miliseconds in the created_at date ( and any other timestamps). Unfortunatly when I specify the format to be Y-m-d H:i:s for the laravel model, it will not ignore the decimals. This is why I'm now looking for ways to default rails to save in the Y-m-d H:i:s format, instead of including the miliseconds. Then both the Rails and Laravel applications would be using the same format and there wouldn't be any conflict.

I know I can go into the DB and change the column type to timestamp(0) which truncates the decimals, but I would perfer to change the format that the frameworks are saving, rather then change what the DB will accept.

Both Y-m-d H:i:s and Y-m-d H:i:s.u are valid timestamps... If I could get Rails to use the Y-m-d H:i:s format or have Laravel ignore the .u when looking at timestamps to prevent the trailing data error, I would be in the clear.

Is there a way to change the default format that rails saves timestamps to the DB?

Is a way to have Larave Carbon ignore the decimal portion of a timestamp in the Y-m-d H:i:s.u format?

Thanks

SegFaultDev
  • 455
  • 1
  • 7
  • 24
  • Rails timestamps are usually stored as `timestamp without time zone` in Postgres. If it's possible to change the types, I would make sure this is the column type you're using. If you have them as that type, I would expect the underlying DB adapters to handle the conversions for you. – Chris Hall Oct 17 '18 at 23:03
  • @ChrisHall hey there. So the current column has a type of `timestamp without time zone`. I could change it to `timestamp(0) without time zone` but would perfer not to make a change like that, as the migration would be fairly big. ~40 tables.... Its hard to believe this is causing such an issue. Im not sure if this is rails or Postgres thats causes these decimal points... – SegFaultDev Oct 18 '18 at 20:06
  • So, I've done a bit more digging and I found [this other question](https://stackoverflow.com/questions/42553178/pdo-get-fractional-seconds-from-postgres) that seems to imply that this kind of thing should "just work" on PHP 7. What version of PHP are you using? – Chris Hall Oct 18 '18 at 21:54
  • @ChrisHall sorry for the deplay in response. I'm using php 7.2.4. Unfortunate I get ether the "trailing data" error or "missing data" error depending on if milliseconds are present or not. The comments for the answer from Masa Sakano has alot more detail on the issue. – SegFaultDev Oct 23 '18 at 14:55

3 Answers3

1

Solution in the Rails side

It seems ActiveRecord used in Rails (5.2) automatically adds decimal seconds down to 1 msec in saving created_at and updated_at or any other Timestamp columns in the DB that accept sub-seconds, as defined in the file active_record/connection_adapters/abstract/quoting.rb

A work around is this. Add this line at a top level in any of the files which would be always read by Rails when accessing a model (such as, ApplicationRecord model file).

Time::DATE_FORMATS[:db] = '%Y-%m-%d %H:%M:%S.000000000'

module ActiveRecord::ConnectionAdapters::Quoting
  alias_method :quoted_date_orig, :quoted_date if ! self.method_defined?(:quoted_date_orig)

  def quoted_date(*rest, **kwd)
    quoted = quoted_date_orig(*rest, **kwd)
    quoted.sub(/(\.\d*)\.\d{6}$/, '\1')
  end
end

You can confirm it from Rails console, after creating a new record,

MyModel.last.created_at.nsec  # => 0

or simply access the DB directly to see it.

Warning

This change affects not only created_at and updated_at but also all the other timestamp columns in the DB. I think you can still save a value to msec (or nsec) precision to such a column by setting a String as opposed to a Time instance to your Model instance like my_model.col_msec_desired = "2018-01-02 03:04:05.678"; then Time::DATE_FORMATS[:db] would not be referenced in saving the record.

Potential solution in the Laravel side

It may be tricky at the time of writing (2018-10-18), but a work seems to be in progress, according to a very recent Laracast post by cmbertsch01

(Note: a major update made a day after from the original post, following the comment.)

Masa Sakano
  • 1,921
  • 20
  • 32
  • Hey Masa, thanks for there reply. Unfortuately I get the following error `(PG::InvalidDatetimeFormat: ERROR: invalid input syntax for type timestamp: "2018-10-18 16:08:33.000000000.203137"` . It does at the 0s but doesnt change it for me. – SegFaultDev Oct 18 '18 at 16:09
  • 1
    @SegFaultDev Which verson of Rails are you using? In Rails 5.2.1 console it works… – Masa Sakano Oct 18 '18 at 21:42
  • 1
    @SegFaultDev Also, which database are you using? I just realised the error you encountered seem to be related to database directly, and so DB-dependent... My test ran well in sqlite3 but I haven't checked it with a more *proper* DB. – Masa Sakano Oct 18 '18 at 22:05
  • 1
    @SegFaultDev I think I have worked it out, and made a major update in my answer. – Masa Sakano Oct 18 '18 at 22:57
  • Hey Masa, I'm using 5.2.0 with a Postgres DB. I believe the issue is from a mixture of rails + postgres. I just read over your edits! I will take a look at the laracast. I believe the solution may have to come form the laravel side, and allow rails + postgres use decimals. – SegFaultDev Oct 19 '18 at 23:33
  • @SegFaultDev My (revised) solution should save timestamps with the sub-second part of zero, which answers the first option of your question. But I don't know if Laravel can correctly read timestamp columns that have a milli-second part in DB. Does it fail even if the value of the part is zero? – I too am interested to know. If you find it and post a comment about it, I'd appreciate it! – Masa Sakano Oct 20 '18 at 04:28
  • 1
    Hey @masa-sakano, I found an article that gives 2 ways to fix the issue. (http://matthewtrask.net/blog/Laravel-Postgres-Timestamps/) . The first way via `public $timestamps = false;` allows the created_at and updated_at timestamps with or without the millisecond part to work but you lose some model-manged timestamps funcationlity. The other way is writing you own `getDateFormat()` function and specifying a format... BUT then all your data needs to be in the format you specify otherwise you may get a data missing error, or get the trailing data error again. – SegFaultDev Oct 22 '18 at 18:37
  • 1
    also, if the timestamp in postgres is put in like `2018-10-10 12:45:38.00000`, it is automically corrected to 2018-10-10 12:45:38. So if you had set the format to 'Y-m-d H:i:s.u' and then had something saved at a time where the milliseconds are .000000, it would turn into a `'Y-m-d H:i:s' format, and cause a `missing data` error to appear. I haven't been able to find a way to support both `'Y-m-d H:i:s.u' and `'Y-m-d H:i:s' without setting the `public $timestamps = false;` in each model. Hopefully in 7.3 like you posted about @masa-sakano, this may be possible. – SegFaultDev Oct 22 '18 at 18:41
  • I've actually decided not to go the route of `public $timestamps = false;`. Removing the automatically handling of timestamps is not worth the trouble. Since both ways can't seem to handle a table with different formats of timestamps, I'm thinking I will go the route of changing my timestamp column type to use `timestamp(0)` which will truncate my timestamps that have milliseconds and ensure that the format will be 'Y-m-d H:i:s'. – SegFaultDev Oct 22 '18 at 20:53
  • 1
    @SegFaultDev Thank you for your follow-ups! Most informative. I believe the best strategy of fundamental fix is to correct Laravel so it would not mess up the timestamps (though my answer is from the Ruby side), but it sounds tricky in the current version. `timestamp(0)` would surely work. A drawback is you have to be careful every time you create a new table in DB. You may want to have a look at Rais' code `quoting.rb` (see my answer for the path); it basically modifies the time format, depending on the SQL table format. Ideally, `getDateFormat()` should implement something similar. – Masa Sakano Oct 22 '18 at 22:24
  • Ya I agree with you about the drawback to `timestamp(0)`. I wonder if there is a way to default rails migration `t.timestamps` to use `timestamp(0)`. Currently both my Rails application and Laravel application use the same DB, but I only run migrations via Rails. I will look into fixing the timestamps via Rails code like you have put in your answer. The last place this fails is when manually using sql statements for insertes, if you are not careful the created_at and updated_at will default to include milliseconds. – SegFaultDev Oct 23 '18 at 14:52
  • 1
    Just for anyone one looking at this in the future and wanting to also use a timestamp(0) technique but also want to change the native data type for timestamp to use timestamp(0), i've made another stackoverflow question regarding that. https://stackoverflow.com/questions/52953885/how-to-change-rails-migration-t-timestamps-to-use-timestamp0-without-timezone – SegFaultDev Oct 23 '18 at 16:35
0

In Laravel/PHP, you can convert DateTime with millisecond to without millisecond using below function:

$date = date('Y-m-d H:i:s.u');
echo date('Y-m-d H:i:s', strtotime($date));

In Rails, you can update your time object before sending it to the database, as mentioned below:

time.change(:usec => 0)
  • Hey Yasir, Thanks for the response. While these would work, I would have to do these for each transation to the DB. Im hoping to find somethign that will prevent rails from saving decimals to DB and use `%Y-%m-%d %H:%M:%S` format, or have laravel ignore if there are decimals in the timestamp( i'm getting the Trailing data error) – SegFaultDev Oct 18 '18 at 16:15
0

you can set Time::DATE_FORMATS[:db] in application.rb

Time::DATE_FORMATS[:db] = "Y-m-d H:i:s.u"

looks more at strftime format

Jame
  • 34
  • 2
  • Hey @Jame, thanks for the response. I tried your approached. While it does let me change the format, the decimal still appears. Even if I do `%Y-%m-%d %H:%M` and remove the seconds, the result is `2018-10-18 16:07.812044`... the decimal still gets added for some reason. Any ideas? – SegFaultDev Oct 18 '18 at 16:08