2

I'm trying to achieve the same amount of precision (number of digits after decimal point) for results coming from a SELECT that does some equations.

  1. Is there any difference in terms of performance, good practice of use in case of achieving wanted precision between CAST() or ROUND() besides rounding by the last one? Is there any better alternative?

For the sake of simplicity values in all examples below are hardcoded, they may or may not come from table's columns

I'm using MySQL 8.

If you run example E1 :

-- E1
SELECT (41/99);
-- 0.4141

you'll get 4 digits after decimal point.

  1. Is there any MySQL setting that can bring higher precision right out of the box so I don't need to use:

    -- E2 SELECT ROUND((41/99), 20); -- 0.41414141400000000000

    -- or

    SELECT CAST((41/99) AS DECIMAL(21,20)); -- 0.41414141400000000000

  2. How to get more decimal points precision from E2 in case data used for calculation is int?

If you supply data with decimal point you get much higher precision:

-- E3
SELECT ROUND((41.0/99.0), 20);
-- 0.41414141414141414100

SELECT CAST((41.0/99.0) AS DECIMAL(21,20));
-- 0.41414141414141414100

It is important to me to avoid floating point datatypes due to their approximation of decimals. If the data be coming from table column the column will be decimal datatype. But data for calculations may be also hardcoded.

user207421
  • 305,947
  • 44
  • 307
  • 483
Jimmix
  • 5,644
  • 6
  • 44
  • 71
  • 1
    Possible duplicate of [How MySQL does the math calculation of floating point addition?](https://stackoverflow.com/questions/55309494/how-mysql-does-the-math-calculation-of-floating-point-addition) – Raymond Nijland Mar 24 '19 at 23:17
  • 1
    *" Is there any MySQL setting that can bring higher precision right out of the box so I don't need to use:"* No will have to need to use implicit datatypes in the tables or a implicit cast nin the select like the SQL standards define to get higher precision .. If there is i never came acros this higher precision setting in MySQL. – Raymond Nijland Mar 24 '19 at 23:20
  • Also i believe that `SELECT (41/99)` should be `0` by SQL standards but MySQL and MariaDB allowes this, in some database like PostgreSQL, SQL Server it will give `0` so `SELECT (41/99);` is not portable. – Raymond Nijland Mar 24 '19 at 23:24
  • @Raymond Nijland the question isn't about floating point but about achieving fixed precision of decimals therefore it is not a duplicate. – Jimmix Mar 24 '19 at 23:36
  • @RaymondNijland `SELECT (41/99)` is not portable due to using two ints that lead result to be rounded to another int so result is `0`? Is the output result strictly depending on input datatype? I need to have at least one (or two?) decimals of floats to get a decimal as result? Is that right? – Jimmix Mar 24 '19 at 23:41
  • the duplicated answer will explain how `SELECT (41/99);` will handle like you have executed `SELECT CAST((41/99) AS DECIMAL(4, 4))` in MySQL.. – Raymond Nijland Mar 24 '19 at 23:42
  • *"Is the output result strictly depending on input datatype?"* Yes it should be SQL standard wise. *"I need to have at least one (or two?) decimals of floats to get a decimal as result? Is that right?"* Yes `SELECT (41/99.00)`, `SELECT (41.00/99)` or `SELECT (41.00/99.00)` is portable, but the precision is implementation-defined – Raymond Nijland Mar 24 '19 at 23:45
  • @RaymondNijland thank you for clarification. I think I already have something that works well: `SELECT (CAST(41 AS DECIMAL(22,20)) / CAST(99 AS DECIMAL(22,20)));` gave me: `0.414141414141414141414141` so even better precision than `E3` while `SELECT (ROUND(41,20) / ROUND(99,20));` gave `0.4141` and `SELECT ROUND(ROUND(41,20) / ROUND(99,20), 20);` gave `0.41414141400000000000` so it seems `CAST()` is clear winner. – Jimmix Mar 24 '19 at 23:53
  • But in order to get really 20 digits of precision I needed to wrap whole `CAST()/CAST()` with another one :) `SELECT CAST((CAST(41 AS DECIMAL(22,20)) / CAST(99 AS DECIMAL(22,20))) AS DECIMAL(22,20));` brings wanted 20 digits of precision after decimal point. Same effect if `ROUND` used `ROUND(CAST()/CAST(),20)` – Jimmix Mar 25 '19 at 00:09
  • 1
    seams there is a MySQL setting i just found it for the first first [div_precision_increment](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_div_precision_increment) – Raymond Nijland Mar 25 '19 at 00:11

1 Answers1

4

MySQL uses 64-bit IEEE-754 floating point (aka DOUBLE) for its internal computations unless you specifically get it to use integer or decimal arithmetic by casting your constants. (So does Javascript.)

When displaying numbers it does its best to render them in decimal as accurately as possible. With DOUBLE, that requires a conversion to decimal. If you don't want the default rendering accuracy, but rather want to control it yourself, you may use the ROUND() or TRUNCATE() functions. That's how you control it.

There's not much performance penalty for using these functions. Many programmers who use plain SQL (rather than SQL retrieved by an application program in Java, PHP, or another language) always use one of the functions to retain control over the rendering.

Test carefully if you're depending on DECIMAL-data-type accuracy in computations: MySQL really wants to do them in DOUBLE. Or, better yet, use a strongly typed language to gain precise control over your arithmetic.

O. Jones
  • 103,626
  • 17
  • 118
  • 172
  • you've scared me: "Test carefully if you're depending on DECIMAL-data-type accuracy in computations: MySQL really wants to do them in DOUBLE". But if I use CAST with AS DECIMAL everywhere in SELECT where I operate at data and everywhere before INSERT I am covered and no surprise should happen or not? – Jimmix Mar 25 '19 at 00:13
  • With respect, I intended to scare you. Be *very* careful with this arithmetic-in-SQL application. You *may* be OK. But you should test cases like `10000000000000 + 1` to convince yourself you are not losing precision. If precision is mission-critical you really should do this stuff in a strongly typed language. – O. Jones Mar 25 '19 at 00:32
  • By the way i found something else if MySQL would use `IEEE-754 floating point (aka DOUBLE) ` internal why `SELECT (0.1 + 0.3) = 0.4` results in `1` it should result in `0` if MySQL would? MySQL seams to be defaulting to [Precision Math](https://dev.mysql.com/doc/refman/8.0/en/precision-math-examples.html) (a.k.a DECIMAL-data-type accuracy) where possible – Raymond Nijland Mar 25 '19 at 00:33