3

I want to load data from the last n days from a data source. To do this, I have a project parameter "number_of_days". I use the parameter in an OleDB data source with a SQL Command, with a clause

WHERE StartDate >= CAST(GETDATE() -? as date) 

This parameter is mapped to a project parameter, an Int32. But, if I want to load the last 10 days, it is only giving me the last 8 days.

Version info:

  • SQL Server Data Tools 15.1.61710.120
  • Server is SQL Server 2017 standard edition.

I set up a test package, with as little data as possible. There is this data source:

data source

Parameter:

parameter

Parameter mapping:

parameter mapping

The T-SQL expression (wrong result):

CAST(GETDATE() -? as date)

The SSIS expression for date_calc (correct result):

(DT_DBTIMESTAMP) (DT_DBDATE) DATEADD("DD", - @[$Project::number_of_days]  , GETDATE())

I would think that the T-SQL expression and the SSIS expression give the same result (today minus 10 days) but that is not the case when I run the package and store the results in a table. See column date_diff, which gives 8 days instead of 10:

result

If I replace the parameter by the actual value, I do get the correct result.

A data viewer also shows the incorrect date. When I deploy the package, I get the same result as from the debugger.

Is this a bug, or am I missing something here?

Hadi
  • 36,233
  • 13
  • 65
  • 124
PeterDeV
  • 203
  • 3
  • 10
  • 1
    You can try DateAdd(‘day’,-?,getdate()) instead off getdate - ? – Sanpas Jan 21 '19 at 12:08
  • @Pascal Sanchez That does work correctly... But I wonder why that gives me a different result, when SELECT DATEADD(day, -10, getdate()) and SELECT GETDATE()-10 are the same when run in SSMS. – PeterDeV Jan 21 '19 at 12:16
  • According [this documentation](https://learn.microsoft.com/en-us/sql/integration-services/map-query-parameters-to-variables-in-an-execute-sql-task) for an OleDb the parameter names should be 0,1,2,... ? – LukStorms Jan 21 '19 at 13:00
  • @LukStorms Thanks, I fixed that, but it doesn't change the results. Seems to work identical with Parameter0, etc. instead of 0, 1, 2. – PeterDeV Jan 21 '19 at 13:09
  • 2
    What if you simplify the test, just to see what's really put in those variables. `select ? as v0, ? as v1, ? as v2;` – LukStorms Jan 21 '19 at 13:15
  • @LukStorms, that is what the first value (param_value) does. `select ? as param_Value`doesn't parse, but `select (cast ? as int) as param_value` does, which gives the correct value, i.e. 10 in the example. So that leads me to the conclusion that the parameter itself is correctly inserted, but the calculation with `getdate() -?` is wrong. There is only one parameter. The variable date_calc uses that same parameter in an expression. – PeterDeV Jan 21 '19 at 14:04
  • So that casting is needed. Oh well. But I have some doubt that the problem is with a subtraction from getdate. Would this be correct: `select p0, p1, p2, cast(dateadd(day,-p1, getdate()) as date) as prev_date from (select cast(? as int) as p0, cast(? as int) as p1, cast(? as date) as p2) as params`. – LukStorms Jan 21 '19 at 14:32
  • All that works correctly. So, `cast(getdate() - p1 as date)` works, but `cast(getdate() - ? as date)` does not. Strangely enough, `cast(getdate() - cast(? as int) as date)` also works correctly. – PeterDeV Jan 21 '19 at 15:55
  • @LukStorms the main problem is in the difference of datetime base value defined in ssis and sql server database engine. You can check my answer for more info – Hadi Jan 27 '19 at 10:33

1 Answers1

3

I think the main problem is how OLEDB source detect the parameter data type, i didn't find an official documentation that mentioned that, but you can do a small experiment to see this:

Try to write the following Query in the SQL Command in the OLEDB Source:

SELECT ? as Column1

And then try to parse the query, you will get the following error:

The parameter type for '@P1' cannot be uniquely deduced; two possibilities are 'sql_variant' and 'xml'.

Which means that the query parser try to figure out what is the data type of these parameter, it is not related to the variable data type that you have mapped to it.

Then try to write the following query:

SELECT CAST(? AS INT) AS Column1

And then try to parse the query, you will get:

The SQL Statement was successfully parsed.


Now, let's apply these experiment to your query:

Try SELECT CAST(GETDATE() - ? AS DATE) as Column1 and you will get a wrong value, then try SELECT CAST(GETDATE() - CAST(? AS INT) AS DATE) AS Column1 and you will get a correct value.

Update 1 - Info from official documentation

From the following OLEDB Source - Documentation:

The parameters are mapped to variables that provide the parameter values at run time. The variables are typically user-defined variables, although you can also use the system variables that Integration Services provides. If you use user-defined variables, make sure that you set the data type to a type that is compatible with the data type of the column that the mapped parameter references.

Which implies that the parameter datatype is not related to the variable data type.


Update 2 - Experiments using SQL Profiler

As experiments, i created an SSIS package that export data from OLEDB Source to Recordset Destination. The Data source is the result of the following query:

SELECT *
FROM dbo.DatabaseLog
WHERE PostTime < CAST(GETDATE() - ? as date)

And The Parameter ? is mapped to a Variable of type Int32 and has the Value 10

Before executing the package, i started and SQL Profiler Trace on the SQL Server Instance, after executing the package the following queries are recorded into the trace:

exec [sys].sp_describe_undeclared_parameters N'SELECT *
FROM dbo.DatabaseLog
WHERE PostTime < CAST(GETDATE() -@P1 as date)'

declare @p1 int
set @p1=1
exec sp_prepare @p1 output,N'@P1 datetime',N'SELECT *
FROM dbo.DatabaseLog
WHERE PostTime < CAST(GETDATE() -@P1 as date)',1
select @p1

exec sp_execute 1,'1900-01-09 00:00:00'

exec sp_unprepare 1

The first command exec [sys].sp_describe_undeclared_parameters is to describe the parameter type, if we run it separately it returns the following information:

enter image description here

It shows that the parameter data type is considered as datetime.

The other commands shows some weird statement:

  • First, the value of @P1 is set to 1
  • The final query is executed with the following value 1900-01-09 00:00:00

Discussion

In SQL Server database engine the base datetime value is 1900-01-01 00:00:00 which can be retrieved by executing the folloing query:

declare @dt datetime
set @dt = 0
Select @dt

On the other hand, in SSIS:

A date structure that consists of year, month, day, hour, minute, seconds, and fractional seconds. The fractional seconds have a fixed scale of 7 digits.

The DT_DATE data type is implemented using an 8-byte floating-point number. Days are represented by whole number increments, starting with 30 December 1899, and midnight as time zero. Hour values are expressed as the absolute value of the fractional part of the number. However, a floating point value cannot represent all real values; therefore, there are limits on the range of dates that can be presented in DT_DATE.

On the other hand, DT_DBTIMESTAMP is represented by a structure that internally has individual fields for year, month, day, hours, minutes, seconds, and milliseconds. This data type has larger limits on ranges of the dates it can present.

Based on that, i think that there is a difference between the datetime base value between SSIS date data type (1899-12-30) and the SQL Server datetime (1900-01-01), which leads to a difference in two days when performing an implicit conversion to evaluate the parameter value.


References

Community
  • 1
  • 1
Hadi
  • 36,233
  • 13
  • 65
  • 124
  • 1
    Thanks, this makes sense... I checked what query is actually received by the database and for the parameters that is: `(@P1 int,@P2 int,@P3 date,@P4 datetime)` If you put an extra `cast(? as int)` it will parse an int. So the last one (P4) is parsed as a datetime, which should be an int. That should still not be an issue because `select cast(cast(-10 as datetime) as int)` just gives -10. Even `select getdate() - (cast(10 as datetime))` works fine. So I guess it also sends the wrong datetime parameter value. Unfortunately I am not much of a DBA and not sure how to figure that out. – PeterDeV Jan 22 '19 at 09:18
  • @PeterDeV i don't think that `cast(10 as datetime)` works fine, it throws a data conversion exception – Yahfoufi Jan 22 '19 at 11:26
  • @Yahfoufi In both SQL Server 2012 and 2017 I get a result of '1900-01-11'. Also according to the documentation it should work: https://learn.microsoft.com/en-us/sql/t-sql/data-types/data-type-conversion-database-engine?view=sql-server-2017 – PeterDeV Jan 22 '19 at 12:05
  • @PeterDeV anything new? – Hadi Jan 25 '19 at 11:55
  • @Hadi Nope... I created a workaround and told my colleagues to not use `GETDATE() -?`, but to use dateadd or an extra cast instead. I think it's a bug, but I am not sure where exactly. I would like to see what the actual parameter values that are sent to the DBMS are, but that means I have to set up the Profiler to trace that, and I don't know how to do that, nor do I have the time to learn how to do that. Thanks for the help though! – PeterDeV Jan 25 '19 at 12:11
  • @PeterDeV don't worry about that i will do it. And i will provide the results in few days – Hadi Jan 25 '19 at 12:24
  • @PeterDeV check my answer update, i think that the situation is clear now. – Hadi Jan 26 '19 at 13:13
  • 1
    @Hadi, very interesting, thanks a lot! The take home message definitely is that in SSIS "Days are represented by whole number increments, starting with 30 December 1899", whereas SQL Server itself uses 1900-01-01. That would definitely explain the 2 days difference. – PeterDeV Jan 26 '19 at 14:30
  • 2
    This is an answer worth mentioning if I ever notice another question related to this. Well done. – LukStorms Jan 27 '19 at 10:57