0

EDIT: The answer here: Is floating point math broken? assists in understanding this question. However, this question is not language agnostic. It is specific to the documented behavior and affinity of floating point numbers as handled by SQLite. Having a very similar answer to a different question != duplicate question.

QUESTION: I have a rather complex SQLite Where Clause comparing numerical values. I have read and "think" I understand the Datatype Documentation here: https://www.sqlite.org/datatype3.html

Still confused as to the logic SQLite uses to determine datatypes in comparison clauses such as =, >, <, <> etc. I can narrow my example down to this bit of test SQL of which the results make little sense to me.

SELECT
    CAST(10 AS NUMERIC) + CAST(254.53 AS NUMERIC) = CAST(264.53 AS NUMERIC) AS TestComparison1,

    CAST(10 AS NUMERIC) + CAST(254.54 AS NUMERIC) = CAST(264.54 AS NUMERIC) AS TestComparison2

Result: "1" "0"

The second expression in the select statement (TestComparison2) is converting the left-side of the equation to a TEXT value. I can prove this by casting the right-side of the equation to TEXT and the result = 1.

Obviously I'm missing something in the way SQLite computes Affinity. These are values coming from columns in a large/complex query. Should I be casting both sides of the equations in WHERE/Join Clauses to TEXT to avoid these issues?

maplemale
  • 2,006
  • 21
  • 35
  • How are you issuing that query? Both of those expressions have a `typeof()` integer. – Dan D. Apr 15 '18 at 18:52
  • @DanD. I am executing the query using the latest version of SQLite DB Browser for SQLite 3.10.1 in my test enviroment and using the latest binaries for the latest version of SQLite3.dll in a .net console app (both the same results). I understand from reading the documentation they should be integer. If that is the case, then maybe this is a rounding bug. – maplemale Apr 15 '18 at 19:01
  • @DanD. FYI - only reason i'm doing a CAST of Numeric in my test example is because the columns these come from are Numeric. I know that means nothing to SQLite (it will determine the datatype on the fly using the affinity rules). However, at this point I suspect a bug and am using this as an example that closely simulates my table setup and successfully reproducing what I believe is a bug. However, you can remove all of the casting and it seems to make no difference - issue still presents itself. – maplemale Apr 15 '18 at 19:11
  • It is floating point: 10 + 254.53 is 264.53 but 10 + 254.54 is 264.53999999999996. – Dan D. Apr 15 '18 at 19:16
  • @DanD. Ah... you're right! It is using float as soon as you apply the + operator. This seems to contradict the documentation which states "Any operators applied to column names, including the no-op unary "+" operator, convert the column name into an expression which always has no affinity." Am I just misunderstanding what that statement really means? Thanks for your explanation! I believe in this case, round() functions will solve the issue for me. – maplemale Apr 15 '18 at 20:15

1 Answers1

1

The reason why you are not getting the expected result is that the underlying results will be floating point.

Although DataTypes in SQLite3 covers much, you should also consider the following section from Expressions :-

Affinity of type-name Conversion Processing

NONE

Casting a value to a type-name with no affinity causes the value to be converted into a BLOB. Casting to a BLOB consists of first casting the value to TEXT in the encoding of the database connection, then interpreting the resulting byte sequence as a BLOB instead of as TEXT.

TEXT

To cast a BLOB value to TEXT, the sequence of bytes that make up the BLOB is interpreted as text encoded using the database encoding.

Casting an INTEGER or REAL value into TEXT renders the value as if via sqlite3_snprintf() except that the resulting TEXT uses the encoding of the database connection.

REAL

When casting a BLOB value to a REAL, the value is first converted to TEXT.

When casting a TEXT value to REAL, the longest possible prefix of the value that can be interpreted as a real number is extracted from the TEXT value and the remainder ignored. Any leading spaces in the TEXT value are ignored when converging from TEXT to REAL.

If there is no prefix that can be interpreted as a real number, the result of the conversion is 0.0.

INTEGER

When casting a BLOB value to INTEGER, the value is first converted to TEXT. When casting a TEXT value to INTEGER, the longest possible prefix of the value >that can be interpreted as an integer number is extracted from the TEXT value and the remainder ignored. Any leading spaces in the TEXT value when converting from TEXT to INTEGER are ignored.

If there is no prefix that can be interpreted as an integer number, the result of the conversion is 0.

If the prefix integer is greater than +9223372036854775807 then the result of the cast is exactly +9223372036854775807. Similarly, if the prefix integer is less than -9223372036854775808 then the result of the cast is exactly -9223372036854775808.

When casting to INTEGER, if the text looks like a floating point value with an exponent, the exponent will be ignored because it is no part of the integer prefix. For example, "(CAST '123e+5' AS INTEGER)" results in 123, not in 12300000.

The CAST operator understands decimal integers only — conversion of hexadecimal integers stops at the "x" in the "0x" prefix of the hexadecimal integer string and thus result of the CAST is always zero.

A cast of a REAL value into an INTEGER results in the integer between the REAL value and zero that is closest to the REAL value. If a REAL is greater than the greatest possible signed integer (+9223372036854775807) then the result is the greatest possible signed integer and if the REAL is less than the least possible signed integer (-9223372036854775808) then the result is the least possible signed integer.

Prior to SQLite version 3.8.2 (2013-12-06), casting a REAL value greater than +9223372036854775807.0 into an integer resulted in the most negative integer, -9223372036854775808. This behavior was meant to emulate the behavior of x86/x64 hardware when doing the equivalent cast.

NUMERIC

Casting a TEXT or BLOB value into NUMERIC first does a forced conversion into REAL but then further converts the result into INTEGER if and only if the conversion from REAL to INTEGER is lossless and reversible. This is the only context in SQLite where the NUMERIC and INTEGER affinities behave differently.

Casting a REAL or INTEGER value to NUMERIC is a no-op, even if a real value could be losslessly converted to an integer.

NOTE

Before this section there is a section on Literal Values (i.e. casting probably only needs to be applied to values extracted from columns).

Try :-

SELECT
    round(CAST(10 AS NUMERIC) + CAST(254.53 AS NUMERIC),2) = round(CAST(264.53 AS NUMERIC),2) AS TestComparison1,

    round(CAST(10 AS NUMERIC) + CAST(254.54 AS NUMERIC),2) = round(CAST(264.54 AS NUMERIC),2) AS TestComparison2

:- enter image description here

MikeT
  • 51,415
  • 16
  • 49
  • 68
  • yup - expand your answer to explain why and how this fits / doesn't fit with the documentation and I'll mark it as the accepted. Personally, my reading of the documentation means this is a bug or just bad documentation. Or, my lack of reading comprehension. I'm not sure which! :) I'm more of a SQL Server guy and MSSQL uses a much stricter / predicable set of affinity rules. – maplemale Apr 15 '18 at 20:20
  • Thank you! Nice job! That last sentence / copy from the documentation is key to understanding SQLite's behavior in this case. – maplemale Apr 15 '18 at 21:01