R issue with rounding milliseconds

Question

Given the following issue with rounding milliseconds under R. How do I get around it so that the times are correct?

> options(digits.secs=3)
> as.POSIXlt("13:29:56.061", format='%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.060 UTC"
> as.POSIXlt("13:29:56.062", format='%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.061 UTC"
> as.POSIXlt("13:29:56.063", format='%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.063 UTC"

I noticed that this URL provides background information but doesn't solve my issue: Milliseconds puzzle when calling strptime in R.

Also this URL touches on the issue but doesn't solve it: R xts: .001 millisecond in index.

In these cases I do see the following:

> x <- as.POSIXlt("13:29:56.061", format='%H:%M:%OS', tz='UTC')
> print(as.numeric(x), digits=20)
[1] 1339075796.0610001087

The URL also seems to indicate that this is just a display issue but I've noticed that using statements like "%OS3" without the options line don't seem to pickup the correct number of digits.

The version I'm using is 32 bit 2.15.0 under Windows but this seems to exist under other situations for R.

Note that my original data is these date time strings within a CSV file I must find a way of converting them into the correct millisecond time from a string.

The use of format() here is unnecessary and distracting . . . — mdsumner, Jun 07 '12 at 12:59
print(56.061, digits=20) #[1] 56.060999999999999943; You can add 0.0001 seconds to all your times and get the corrected truncation. The precision of the floating point numbers should be the same on 32 and 64 bit machines. — IRTFM, Jun 07 '12 at 13:37

Gavin Simpson · Accepted Answer · 2012-06-07T13:16:15.960

I don't see that:

> options(digits.secs = 4)
> as.POSIXlt("13:29:56.061", format = '%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.061 UTC"
> as.POSIXlt("13:29:56.062", format = '%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.062 UTC"
> as.POSIXlt("13:29:56.063", format = '%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.063 UTC"
> options(digits.secs = 3)
> as.POSIXlt("13:29:56.061", format = '%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.061 UTC"
> as.POSIXlt("13:29:56.062", format = '%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.062 UTC"
> as.POSIXlt("13:29:56.063", format = '%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.063 UTC"

with

> sessionInfo()
R version 2.15.0 Patched (2012-04-14 r59019)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_GB.utf8       LC_NUMERIC=C             
 [3] LC_TIME=en_GB.utf8        LC_COLLATE=en_GB.utf8    
 [5] LC_MONETARY=en_GB.utf8    LC_MESSAGES=en_GB.utf8   
 [7] LC_PAPER=C                LC_NAME=C                
 [9] LC_ADDRESS=C              LC_TELEPHONE=C           
[11] LC_MEASUREMENT=en_GB.utf8 LC_IDENTIFICATION=C      

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base

With the "%OSn" format strings, one forces truncation. If the fractional second cannot be represented exactly in floating points then the truncation may very well go the wrong way. If you see things going to wrong way you can also round explicitly to the unit you want or add a half of the fraction you wish to operate at (in the case shown 0.0005):

> t1 <- as.POSIXlt("13:29:56.061", format = '%H:%M:%OS', tz='UTC')
> t1
[1] "2012-06-07 13:29:56.061 UTC"
> t1 + 0.0005
[1] "2012-06-07 13:29:56.061 UTC"

(but a I said, I don't see the problem here.)

This latter point was made by Simon Urbanek on the R-Devel mailing list on 30-May-2012.

@AndrewStern I can't I don't have a 32-bit system to try it on. I've updated my Answer. Try to add a small fraction (0.0005 in your case) to your times *after* you have got them as `"POSIXlt"` objects and see if that improves the situation. Follow that R-Devel thread for more details. — Gavin Simpson, Jun 07 '12 at 13:21
I can reproduce - I have both 32bit and 64bit R installed on a Win7 64bit system. It seems the problem is specific to 32bit R. — Fhnuzoag, Jun 07 '12 at 13:24
@Fhnuzoag Ok, so this is a floating point issue. Andrew should try adding the small fraction to times that are being truncated. — Gavin Simpson, Jun 07 '12 at 13:39
Adding 0.0005 to my vector before printing it out seems to have done the trick. — Andrew Stern, Jun 07 '12 at 17:58

score 3 · Answer 2 · edited May 23 '17 at 12:19

This is the same problem as Milliseconds puzzle when calling strptime in R.

Your example:

> x <- as.POSIXlt("13:29:56.061", format='%H:%M:%OS', tz='UTC')
> print(as.numeric(x), digits=20)
[1] 1339075796.0610001087

is not representative of the problem. as.numeric(x) converts your POSIXlt object to POSIXct before converting to numeric, so you get different floating-point-precision rounding errors.

That's not how print.POSIXlt (which calls format.POSIXlt) works. format.POSIXlt formats each element of the POSIXlt list construct individually, so you would need to look at:

print(x$sec, digits=20)
[1] 56.060999999999999943

And that number is truncated at the third decimal place, so you see 56.060. You can see this by calling format directly:

> format(x, "%H:%M:%OS6")
[1] "13:29:56.060999"

score 3 · Answer 3 · answered Oct 16 '15 at 00:59

In testing I have noted that this issue still exists for 32bit R 3.01 and that this is due to a truncation of floating point data that is specific to the 32bit implementation of print, format and as.character operators for POSIXlt date times.

The underlying data hasn't been stored in a different type that is leading to the truncation in one case (32bit) and not the other (64bit), but the "print", "format" and "as.character" functions for the POSIXlt type specifically which are used to display the POSIXlt data as a displayable string.

Whilst the documented behaviour is that these functions truncate (ignore) extra digits (as mentioned by @Gavin Simpson), this is not true in the same way for 32 and 64 bit versions. To demonstrate; we'll generate 1000 different times and perform some comparisons operations:

> options(digits.sec=3)
> x = as.POSIXlt("13:29:56.061", format='%H:%M:%OS', tz='UTC')

> for (i in 0:999) {
>     x[i+1] = as.POSIXlt(paste0("13:29:56.",sprintf("%03d",i)),format='%H:%M:%OS',tz='UTC')
> }

> sum(x[2:1000]>x[1:999])
[1] 999

Under both 32 bit and 64 bit the comparison operators are consistent, however under 32 bit I see:

> x[1:6]
[1] "2015-10-16 13:29:56.000 UTC" "2015-10-16 13:29:56.000 UTC"
[3] "2015-10-16 13:29:56.002 UTC" "2015-10-16 13:29:56.003 UTC"
[5] "2015-10-16 13:29:56.003 UTC" "2015-10-16 13:29:56.005 UTC"

So it is clearly a display issue. Looking at the actual numbers in the POSIXlt datatype, particularly the seconds we can see what appears to happen:

> y = (x[1:6]$sec) 
> y
[1] 56.000 56.001 56.002 56.003 56.004 56.005
> trunc(y*1000)/1000
[1] 56.000 56.001 56.002 56.003 56.004 56.005
> trunc((y-floor(y))*1000)/1000
[1] 0.000 0.000 0.002 0.003 0.003 0.005

I would suggest that this is a bug that should be fixed in the underlying base library, as a temporary fix though, you can overwrite the "print", "as.character" and "format" functions to change the output to your desired output e.g.

format.POSIXlt = function(posix) {
    return(paste0(posix$year+1900,"-",sprintf("%02d",posix$mon+1),"-",sprintf("%02d",posix$mday)," ",
        sprintf("%02d",posix$hour),":",sprintf("%02d",posix$min),":",sprintf("%002.003f",posix$sec)))
    }

print.POSIXlt = function(posix) {
    print(paste0(posix$year+1900,"-",sprintf("%02d",posix$mon+1),"-",sprintf("%02d",posix$mday)," ",
        sprintf("%02d",posix$hour),":",sprintf("%02d",posix$min),":",sprintf("%002.003f",posix$sec)))
    }

as.character.POSIXlt = function(posix) {
    return(paste0(posix$year+1900,"-",sprintf("%02d",posix$mon+1),"-",sprintf("%02d",posix$mday)," ",
        sprintf("%02d",posix$hour),":",sprintf("%02d",posix$min),":",sprintf("%002.003f",posix$sec)))
    }

score 1 · Answer 4 · answered Jun 07 '12 at 12:55

1

The milliseconds are there:

 unclass(as.POSIXlt("13:29:56.061", '%H:%M:%OS', tz='UTC'))
 $sec
 [1] 56.061
 ...

(There's no need to call format here, it's the name of an argument not the required input from some other function).

Otherwise, I cannot reproduce (on Windows 64-bit R 2.15.0):

options(digits.secs = 3)
as.POSIXlt("13:29:56.061", '%H:%M:%OS', tz='UTC')
[1] "2012-06-07 13:29:56.061 UTC"

sessionInfo()
R version 2.15.0 Patched (2012-05-05 r59321)
Platform: x86_64-pc-mingw32/x64 (64-bit)
...

answered Jun 07 '12 at 12:55

mdsumner

29,099
6
83
91

It does seem to be correct when I unclass it using the following: unclass(as.POSIXlt("13:29:56.061", "%H:%M:%OS", tz='UTC')) but the screen still shows the incorrect milliseconds when displaying using : as.POSIXlt("13:29:56.061", "%H:%M:%OS", tz='UTC') . Note that I'm on the 32bit version and the 64bit version might be more accurate since the registers are bigger. – Andrew Stern Jun 07 '12 at 13:03

R issue with rounding milliseconds

4 Answers4

Linked

Related