SQL transformed result

Question

I have data in DB like this

INPUT

Id  |   Start Date  |   End Date    |   Value
1       1/2/2010        1/6/2010        20
2       1/5/2010        1/7/2010        80

I need to transform this data using sql query like this

OUTPUT

Id  |   Month       |   Value
1       1/2/2010        20
1       1/3/2010        20
1       1/4/2010        20
1       1/5/2010        20
1       1/6/2010        20
2       1/5/2010        80
2       1/6/2010        80
2       1/7/2010        80

Please suggest possible solution to take this. Consider performance as input table has millions of records and we need to process all. I am using SQL 2008, and want to avoid loop or cursors. Thanks.

score 1 · Accepted Answer · answered Mar 04 '11 at 08:36

Have a look at using a recursive CTE query then.

Something like

DECLARE @Table TABLE(
    Id INT,
    StartDate DATETIME,
    EndDate DATETIME,
    Value FLOAT
)

INSERT INTO @Table SELECT 1,'1/2/2010','1/6/2010',20
INSERT INTO @Table SELECT 2,'1/5/2010','1/7/2010',80

;WITH Vals AS (
        SELECT  id,
                StartDate,
                EndDate,
                Value
        FROM    @Table
        UNION   ALL
        SELECT  id,
                StartDate + 1,
                EndDate,
                Value
        FROM    Vals
        WHERE   StartDate + 1 <= EndDate
)
SELECT  *
FROM    Vals
ORDER BY    id,
            StartDate
OPTION (MAXRECURSION 0)

Perfect, but performance is very poor. I have 113000 recrds, and in half an hour it has reached to 11,000 th records so far. I have another solution in CLR Integration project, which is yet better (but unacceptable). Performance is primary concern why I am looking for other options. — hungryMind, Mar 04 '11 at 09:13

Ekkehard.Horner · Answer 2 · 2011-03-04T17:17:22.523

Create a helper table Beta:

 CREATE TABLE Beta (
 Monat DATETIME,
 )

fill it with all monthes you need:

 INSERT INTO Beta VALUES( '1/1/2010' )
 ...
 INSERT INTO Beta VALUES( '1/12/2010' )  ' maybe '12/1/2010'

and:

 SELECT A.Id, B.Monat, A.Value
 FROM  <YourTable> A, Beta B
 WHERE B.Monat >= A.StartDate And B.Monat <= A.EndDate

(no garanties wrt performance)

I did not anticipate that something like this has to be 'proven' -

    ===============================================================================
    SO5192555 - select for hungryMind
    -------------------------------------------------------------------------------
    -------------------------------------------------------------------------------
    SELECT * FROM Alpha
    -------------------------------------------------------------------------------
    |Id|StartDate|EndDate |Value|
    | 1| 2/1/2010|6/1/2010|   20|
    | 2| 5/1/2010|7/1/2010|   80|
    -------------------------------------------------------------------------------
    -------------------------------------------------------------------------------
    SELECT A.Id, B.Monat, A.Value FROM  Alpha A, Beta B WHERE B.Monat >= A.StartDate And B.Monat <= A.EndDate
    -------------------------------------------------------------------------------
    |Id|Monat   |Value|
    | 1|2/1/2010|   20|
    | 1|3/1/2010|   20|
    | 1|4/1/2010|   20|
    | 1|5/1/2010|   20|
    | 1|6/1/2010|   20|
    | 2|5/1/2010|   80|
    | 2|6/1/2010|   80|
    | 2|7/1/2010|   80|
    ===============================================================================
    xpladolib.vbs: Erfolgreich beendet. (0) [ 0.17969 secs ]

I cannot create table beta. I have several rows (around a million), I group them to get max and min dates and insert into table beta, and perform query on join, does not seem will give better performance. — hungryMind, Mar 04 '11 at 18:29

score 0 · Answer 3 · answered Mar 04 '11 at 22:04

Despite hungryMind's "I can't" and "it seems", I still think that using the helper table is the right way to go:

Filling table Alpha with 50.000 records like

SELECT TOP 5 *, DATEDIFF("d", StartDate, EndDate ) + 1 AS Days FROM Alpha ORDER BY Id
-------------------------------------------------------------------------------
|Id|StartDate |EndDate   |Value|Days|
| 1| 12/6/2001| 5/15/2002|   10| 161|
| 2| 8/2/2001 |10/27/2001|   20|  87|
| 3|10/28/2000| 6/17/2001|   30| 233|
| 4| 1/15/2000| 8/30/2000|   40| 229|
| 5| 3/25/2002|10/23/2002|   50| 213|
-------------------------------------------------------------------------------
SELECT TOP 5 *, DATEDIFF("d", StartDate, EndDate ) + 1 AS Days FROM Alpha ORDER BY Id DESC
-------------------------------------------------------------------------------
|Id   |StartDate |EndDate   |Value |Days|
|50000|10/31/2001| 5/6/2002 |500000| 188|
|49999| 8/31/2002|12/31/2002|499990| 123|
|49998| 4/11/2002|11/11/2002|499980| 215|
|49997| 3/13/2002|12/16/2002|499970| 279|
|49996| 7/4/2002 | 7/27/2002|499960|  24|

and creating the helper table Beta based on the range MIN( StartDate ) ... MAX( EndDate ) - I used a loop to insert all the 1297 days -

-----------------------------------------------
SELECT TOP 5 * FROM Beta ORDER BY Monat
-----------------------------------------------
|Monat   |
|1/1/2000|
|1/2/2000|
|1/3/2000|
|1/4/2000|
|1/5/2000|
-----------------------------------------------
SELECT TOP 5 * FROM Beta ORDER BY Monat DESC
-----------------------------------------------
|Monat    |
|7/20/2003|
|7/19/2003|
|7/18/2003|
|7/17/2003|
|7/16/2003|

and executing

SELECT A.Id, B.Monat, A.Value
INTO Gamma FROM  Alpha A, Beta B
WHERE B.Monat >= A.StartDate And B.Monat <= A.E

to insert 7.522.243 records into table Gamma:

SELECT TOP 5 * FROM Gamma Order BY Id, Monat
-----------------------------------------------------------
|Id|Monat     |Value|
| 1| 12/6/2001|   10|   <--- | 1| 12/6/2001| 5/15/2002|   10| 161|
| 1| 12/7/2001|   10|
| 1| 12/8/2001|   10|
| 1| 12/9/2001|   10|
| 1|12/10/2001|   10|
-----------------------------------------------------------
SELECT TOP 5 * FROM Gamma Order BY Id DESC, Monat
-----------------------------------------------------------
|Id   |Monat     |Value |
|50000|10/31/2001|500000| <---- |50000|10/31/2001| 5/6/2002 |500000| 188|
|50000| 11/1/2001|500000|
|50000| 11/2/2001|500000|
|50000| 11/3/2001|500000|
|50000| 11/4/2001|500000|

took about 2 mins on my WinXP/SQLExpress/1 GB Mem/VirtualBox machine. Doing just the "SELECT INTO" took 26 secs.

50.000 source records are less than 113000 or 'millions', but I did my test using ADO/OleDB via VBScript. Surely a MS Server Admin can do better.

I will try this approach as well. Thanks anyway. – hungryMind Mar 05 '11 at 09:17 — hungryMind, Mar 05 '11 at 09:17

SQL transformed result

3 Answers3