How to deal with the problem of insufficient precision of lua floating point numbers

Question

When using lua to handle floating point numbers I found that lua can handle very limited precision, for example:

print(3.14159265358979)

output:

3.1415926535898

The result will be missing a few decimal places, which will lead to calculation bias. How can I deal with such a lack of precision

You can read about how numbers are represented in LUA [here](https://www.lua.org/pil/2.3.html). And [this question](https://stackoverflow.com/questions/45270543/dealing-with-big-numbers-in-lua) has some answers with tips on LUA modules that can let you use numbers with more precision. — Jason Goemaat, Nov 03 '21 at 04:16
"*very limited precision*" I would not call 12 decimal places "very limited precision". — Nicol Bolas, Nov 03 '21 at 07:15

score 0 · Accepted Answer · answered Nov 03 '21 at 15:04

By default, Lua only displays 14 digits of a number. A float can require 15 to 17 digits to be represented exactly as a base-10 string. We can use a loop to find the right number of digits. Note that %g will drop the trailing zeros, so we can start our search at 15 digits, not 1. This is the function I use:

local function floatToString(x)
  for precision = 15, 17 do
    -- Use a 2-layer format to try different precisions with %g.
    local s <const> = ('%%.%dg'):format(precision):format(x)
    -- See if s is an exact representation of x.
    if tonumber(s) == x then
      return s
    end
  end
end

print(floatToString(3.14159265358979))

Output: 3.14159265358979

How to deal with the problem of insufficient precision of lua floating point numbers

1 Answers1