1

When a file is read through fread, the columns may be read as integer64 (correctly so), but when these are multiplied with numeric, they are not upcasted to numeric (as in C++ or integers in R). While this is a documented behavior in bit64 package. But it is not intuitive, when numbers are multiplied etc. integer64 behaves differently compared to integer.

Also, integer64 when divided against integer gives a numeric variable. So the behavior is very bizarre !

Should we then always fread using colClasses = numeric for columns to be used in arithmeric expressions with numeric etc ?


    file contents
    x,y
    111,0.3
    2147483648,0.3

    > d <- fread(file)     
    > print(d$x*d$y)
            x       y
1:        111       0.3
2: 2147483648       0.3

> as.integer64(111) * 8e-2
integer64
[1] 9
> as.integer64(111) * 8 / 1e2
8.88

Similarly, quantiles and other R functions will not behave correctly with integer64. This issue creeps into all classes that use integer64 like nanotime

cryptickey
  • 169
  • 8

1 Answers1

3

This is the documented behaviour of bit64 package, see Arithmetic precision and coercion in ?bit64:

The fact that we introduce 64 bit long long integers – without introducing 128-bit long doubles – creates some subtle challenges

The multiplication operator * coerces its first argument to integer64 but allows its second argument to be also double: the second argument is internaly coerced to 'long double' and the result of the multiplication is returned as integer64

as.integer64(111) * 8e-2
integer64
[1] 9

The division / and power ^ operators also coerce their first argument to integer64 and coerce internally their second argument to 'long double', they return as double

as.integer64(111) * 8 / 1e2
8.88

To avoid this, you could set integer64 parameter of fread to "double". To be used with care as there is an open issue.

Waldi
  • 39,242
  • 6
  • 30
  • 78
  • Are `integer64` intended to be used in arithmetic operations with `numeric`. If so, why ? Why does `fread` read these as `integer64` and not `numeric`, unless stated otherwise ? Why do we not report a warning when such coercion happen, so one makes a conscious choice ? If one does `quantile` of `integer64`, one does not realize if the quantiles may not be correct ? – cryptickey Feb 25 '22 at 09:56
  • 1
    @cryptickey, this is the default value of parameter `integer64` in `fread` : `integer64=getOption("datatable.integer64", "integer64")`. For the why, `data.table` team surely carefully studied pros and contras. – Waldi Feb 25 '22 at 10:04