1

I'm newish to LinqToSQL and the project that I am working on cannot be changed to something else. I am translating some old SQL code to Linq. Not being that hot at linq, I used Linqer to do the translation for me. The query took about 90 seconds to run, so I thought it must be the linqToSQL. However, when I copied the query that the LinqToSQL produced and ran an ExecuteQuery on the datacontext it was super quick as I expected. I've copied the full queries, rather than trying to distil it down, but it looks like the issue is with something LinqToSQL is doing behind the scenes.

To summarise, if I copy the T-SQL created by linq and run

var results = DB.ExecuteQuery<InvoiceBalanceCheckDTO.InvoiceBalanceCheck>(@"T-SQL created by Linq - see below").ToList()

it completes with expected results in about 0.5 seconds. It runs about the same time directly in SSMS. However, if I use the linqToSQL code that creates the T-SQL and do ToList() it takes ages. The result is only 9 records, although without the constraint to check the balance <> 0, there would be around 19,000 records. It's as if it's getting all 19,000 and then checking <> 0 after it's got the records. I have also changed the Linq to project into the class used above, rather than to an anonymous type, but it makes not difference

This is the original SQL :

SELECT InvoiceNum, Max(AccountCode), Sum(AmountInc) AS Balance
FROM 
    (SELECT InvoiceNum, AccountCode, AmountInc From TourBookAccount WHERE AccDetailTypeE IN(20,30) AND InvoiceNum >= 1000 
    UNION ALL 
    SELECT InvoiceNum, '<no matching invoice>' AS AccountCode, AccountInvoiceDetail.AmountInc 
    FROM AccountInvoiceDetail 
        INNER JOIN AccountInvoice ON AccountInvoiceDetail.InvoiceID=AccountInvoice.InvoiceID 
    WHERE AccDetailTypeE IN(20,30) 
    AND InvoiceNum >= 1000 
    ) as t
GROUP BY InvoiceNum 
HAVING (Sum(t.AmountInc)<>0) 
ORDER BY InvoiceNum

and this is the linq

var test =  (from t in
                        (
                            //this gets the TourBookAccount totals
                            from tba in DB.TourBookAccount
                            where
                            detailTypes.Contains(tba.AccDetailTypeE) &&
                            tba.InvoiceNum >= dto.CheckInvoiceNumFrom
                            select new 
                            {
                                InvoiceNum = tba.InvoiceNum,
                                AccountCode = tba.AccountCode,
                                Balance = tba.AmountInc
                            }
                        )
                        .Concat //note that concat, since it's possible that the AccountInvoice record does not actually exist
                        (
                            //this gets the Invoice detail totals.
                            from aid in DB.AccountInvoiceDetail
                            where
                            detailTypes.Contains(aid.AccDetailTypeE) &&
                            aid.AccountInvoice.InvoiceNum >= dto.CheckInvoiceNumFrom &&
                            select new 
                            {
                                InvoiceNum = aid.AccountInvoice.InvoiceNum,
                                AccountCode = "<No Account Records>",
                                Balance = aid.AmountInc
                            }
                        ) 
                group t by t.InvoiceNum into g
                where Convert.ToDecimal(g.Sum(p => p.Balance)) != 0m
                select new 
                {
                    InvoiceNum = g.Key,
                    AccountCode = g.Max(p => p.AccountCode),
                    Balance = g.Sum(p => p.Balance)
                }).ToList();

and this is the T-SQL that the linq produces

  SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
    SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
    FROM (
        SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
        FROM (
            SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
            FROM [dbo].[TourBookAccount] AS [t0]
            WHERE ([t0].[AccDetailTypeE] IN (20, 30)) AND ([t0].[InvoiceNum] >= 1000)
            UNION ALL
            SELECT [t2].[InvoiceNum],'<No Account Records>' AS [value], [t1].[AmountInc]
            FROM [dbo].[AccountInvoiceDetail] AS [t1]
            INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
            WHERE ([t1].[AccDetailTypeE] IN (20, 30)) AND ([t2].[InvoiceNum] >= 1000)
            ) AS [t3]
        ) AS [t4]
    GROUP BY [t4].[InvoiceNum]
    ) AS [t5]
WHERE [t5].[value] <> 0
PabloInNZ
  • 515
  • 4
  • 14
  • Is this query the first query being performed? When the first query executes there is a one time cost per AppDomain that you must do for Initization. – Scott Chamberlain Aug 26 '13 at 04:46
  • Can you run the profiler on the server, and verify that it does the query you got in a single step? – macwier Aug 26 '13 at 05:02
  • Thanks for suggestions. I'll run the profiler on the SQL box and double check that the SQL that VS told me was running (which is where I got the output) is actually what is running. @Scott - no this is not the first query running – PabloInNZ Aug 26 '13 at 20:16
  • We are using SQL Server Express and I haven't got the time to install SSMS 2012 Sp1 at the moment, which I think comes with profiler. I added a log onto the data context though, and that confirmed that the T-SQL being sent is in one hit and matches that shown by intellisense that I posted. A bit short on time, so going to just send the raw SQL for the moment and come back later to try and find the issue. Thanks for the help. – PabloInNZ Aug 26 '13 at 20:40

2 Answers2

2

I would bet money, that the problem is in this line:

where Convert.ToDecimal(g.Sum(p => p.Balance)) != 0m

What is probably happening, is that it can't translate this to SQL and silently tries to get all rows from db to memory, and then do filtering on in memory objects (LINQ to objects) Maybe try to change this to something like:

where g.Sum(p=>.Balance!=0)
macwier
  • 1,063
  • 1
  • 6
  • 19
  • I thought that too, but look at the generated SQL - it does appear to be translating it... – Blorgbeard Aug 26 '13 at 04:48
  • And usually you get a NotSupportedException "[blahblah] method has no translation to SQL" – Blorgbeard Aug 26 '13 at 04:48
  • 1
    Yeah, i know that it happens on the EF. Im wasn't sure about LINQ to SQL. Can you run the profiler on the server, and verify that this is actually the query that's executed in a single step? – macwier Aug 26 '13 at 04:49
  • Look at the SQL - there is no inline 'convert to decimal' occuring at all. So try @Botis suggestion, remove the conversion from the .Net code and see if it improves performance. Once you have verified that, you can work out the correct code to do what you want. – Nick.Mc Aug 26 '13 at 06:06
  • Possibly the source field is already the right datatype anyway, so the `Convert.ToDecimal` is simply ignored. – Blorgbeard Aug 26 '13 at 06:15
  • Thanks for the suggestions. I removed the convert and it made no difference to the speed. I'm on a bit of a tight schedule, so I think I'll have to park this one for a while and go with sending the direct SQL for now until I can dig deeper – PabloInNZ Aug 26 '13 at 20:37
  • @PabloInNZ Only other thing that comes to my head is that its the LINQ to SQL complining process takes that much time. I don't know a way to measure it, or make sure thats it is indeed a problem, you will have to googole around. Something like: http://peterkellner.net/2009/05/06/linq-to-sql-slow-performance-compilequery-critical/ – macwier Aug 27 '13 at 07:46
  • @Botis - thanks.Coincidentally,I just stumbled across a page about compiling LinqToSQL yesterday, so will give that a whirl when I have time [link](http://msdn.microsoft.com/en-us/magazine/ee336024.aspx) – PabloInNZ Aug 28 '13 at 02:05
2

Well, the answer turned out not to be LinqToSQL itself (although possibly the way it creates the query could be blamed) , but the way SQL server handles the query. When I was running the query on the database to check speed (and running the created T=SQL in DB.ExecuteQuery) I had all the variables hardcoded. When I changed it to use the exact sql that Linq produces (i.e. with variables that are substituted) it ran just as slow in SSMS.

Looking at the execution plans of the two, they are quite different. A quick search on SO brought me to this page : Why does a parameterized query produces vastly slower query plan vs non-parameterized query which indicated that the problem was SQL server's "Parameter sniffing".

The culprit turned out to be the "No Account Records" string

For completeness, here is the generated T-SQL that Linq creates. Change @p10 to the actual hardcoded string, and it's back to full speed ! In the end I just removed the line from the linq and set the account code afterwards and all was good.

Thanks @Botis,@Blorgbeard,@ElectricLlama & @Scott for suggestions.

DECLARE @p0 as Int = 20
DECLARE @p1 as Int = 30
DECLARE @p2 as Int = 1000
DECLARE @p3 as Int = 20
DECLARE @p4 as Int = 30
DECLARE @p5 as Int = 1000
DECLARE @p6 as Int = 40
DECLARE @p7 as Int = 10
DECLARE @p8 as Int = 0
DECLARE @p9 as Int = 1
DECLARE @p10 as NVarChar(4000)= '<No Account Records>' /*replace this parameter with the actual text in the SQl and it's way faster.*/
DECLARE @p11 as Decimal(33,4) = 0


SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
    SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
    FROM (
        SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
        FROM (
            SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
            FROM [dbo].[TourBookAccount] AS [t0]
            WHERE ([t0].[AccDetailTypeE] IN (@p0, @p1)) AND ([t0].[InvoiceNum] >= @p2)
            UNION ALL
            SELECT [t2].[InvoiceNum], @p10 AS [value], [t1].[AmountInc]
            FROM [dbo].[AccountInvoiceDetail] AS [t1]
            INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
            WHERE ([t1].[AccDetailTypeE] IN (@p3, @p4)) AND ([t2].[InvoiceNum] >= @p5) AND ([t2].[InvoiceStatusE] <= @p6) AND ([t2].[InvoiceTypeE] = @p7) AND ([t1].[BookNum] <> @p8) AND ([t1].[AccDetailSourceE] = @p9)
            ) AS [t3]
        ) AS [t4]
    GROUP BY [t4].[InvoiceNum]
    ) AS [t5]
WHERE [t5].[value] <> @p11


SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
    SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
    FROM (
        SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
        FROM (
            SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
            FROM [dbo].[TourBookAccount] AS [t0]
            WHERE ([t0].[AccDetailTypeE] IN (20, 30)) AND ([t0].[InvoiceNum] >= 1000)
            UNION ALL
            SELECT [t2].[InvoiceNum], '<No Account Records>' AS [value], [t1].[AmountInc]
            FROM [dbo].[AccountInvoiceDetail] AS [t1]
            INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
            WHERE ([t1].[AccDetailTypeE] IN (20, 30)) AND ([t2].[InvoiceNum] >= 0) AND ([t2].[InvoiceStatusE] <= 40) AND ([t2].[InvoiceTypeE] = 10) AND ([t1].[BookNum] <> 0) AND ([t1].[AccDetailSourceE] = 1)
            ) AS [t3]
        ) AS [t4]
    GROUP BY [t4].[InvoiceNum]
    ) AS [t5]
WHERE [t5].[value] <> 0
Community
  • 1
  • 1
PabloInNZ
  • 515
  • 4
  • 14
  • If you are creating a new project I would recommend you use entity framework as opposed to Linq to SQL. I believe ef does that optimisation automagically. EF will continue to receive better support from MSFT than l2s – Aron Aug 29 '13 at 00:57
  • @Aron - For my own stuff, I actually use Lightspeed [link](http://www.mindscapehq.com/products/lightspeed), but we are pretty locked in to LinqToSQL at work – PabloInNZ Aug 29 '13 at 02:11