0

I have a SQLite database C# application which I'm running in memory (:memory:). My table has about 650k rows, which isn't much. I want to get fast response-times on the following query (syntax fetched from LINQ to SQL dynamic LINQ query)

SELECT 
  SUM([t0].[Value1]) AS [Sum],
  [t0].[Dim1] AS [Primary], 
  [t0].[Dim2] AS [Secondary]
FROM [BudgetLine] AS [t0]
WHERE [t0].[BudgetID] = 4
GROUP BY [t0].[Dim1], [t0].[Dim2]

Which would be the optimal index for this query? My only index other than the primary key looks like the following...

create index IX_0 on budgetLine (BudgetID, Dim1, Dim2)
create index IX_1 on budgetLine (BudgetId)
create index IX_2 on budgetLine (Dim1, Dim2)
create index IX_3 on budgetLine (BudgetID, Dim1, Dim2,Value1)

Currently the execution times varies, but around 1s is the current average. I want this query to take less than 0.5s at least.

The table has about 50 columns.

Please assist

Update: See my 4 indexes above, with these I'm getting 0.8s response...

JohanLarsson
  • 475
  • 1
  • 8
  • 23
  • If [EXPLAIN QUERY PLAN](http://www.sqlite.org/eqp.html) shows a single search with the covering index, there's nothing to improve. – CL. May 29 '14 at 16:04
  • I got "SEARCH TABLE BudgetLine AS t0 USING COVERING INDEX IX_2 (BudgetID=?) (~10 rows)". Are you saying it won't get any better than that? If so then I'm not so impressed by the SQLite performance – JohanLarsson May 29 '14 at 16:07
  • Now all the sudden I'm averaging 1.5s. Totally confused. Yesterday I got better times, however the index I tried might have been slightly different. Are there any nice tools to provide index best guess other than SQLiteBrowser EXPLAIN QUERY PLAN? – JohanLarsson May 29 '14 at 16:26

2 Answers2

0

I would try these indexes.

create index IX_2 on budgetLine(BudgetId)
create index IX_3 on budgetLine(Dim1, Dim2)
Bartosz Wójtowicz
  • 1,321
  • 10
  • 18
  • Ok, now I'm getting 0.8s response times in average using the 4 indexes as described above. Any further improvements? Which indexes are used when? – JohanLarsson May 29 '14 at 18:05
0

Remove indexes

create index IX_3 on budgetLine (BudgetID, Dim1, Dim2,Value1)
create index IX_0 on budgetLine (BudgetID, Dim1, Dim2)

and just use the budgetid index provided by bartosz (but NOT the dim1, dim2)

an in memory database is going to be limited by your systems memory - are you running on an underpowered or overtaxed machine? Is there a reason why you're using :memory: instead of an actual disk file? (see Stack overflow question here)

Also consider reading the SQLite documentation about performance tuning, SQLite performance for windows

Also try changing your SQL query as follows:

    select
    SUM([Value1]) AS [Sum]
    ,[Dim1] AS [Primary], 
    [Dim2] AS [Secondary]
from
(
    SELECT 
      value1, dim1, dim2
    FROM [BudgetLine]
    WHERE [BudgetID] = 4
) as a
GROUP BY [Dim1], [Dim2]
Community
  • 1
  • 1
Nathan
  • 51
  • 3
  • That actually slowed things down. The best performance I've gotten so far is using all four. I am running on my development machine now which has 8GB of RAM, so I'm expecting it to be faster when deployed to a real server.. – JohanLarsson May 29 '14 at 18:20
  • The best performance I've gotten so far is using all four or IX_3 only Will have a look at the documentation – JohanLarsson May 29 '14 at 18:26
  • I've updated my question a bit to get more information about what you're looking for, but I've linked a stack overflow question where :memory: has been discussed quite a bit. You might also try changing your query, (see my edit) – Nathan May 29 '14 at 18:36
  • Thanks. I will try that query. A challenge to do it in Dynamic Linq, but I'll find a solution for sure. We decided to use the :memory: due to that we thought it would be fast. I realize that its just pageing without disk IO we are benefitting from. Question is if there is another memory db suitable with MS .net and LINQ to SQL or Entity Framework. Our server will be loaded with lots of RAM so that shouldnt be a problem, using SQLite then that is – JohanLarsson May 29 '14 at 21:25
  • Sqlite is generally extremely good for memory databases, but I've heard success stories from Firebird as well (http://www.firebirdsql.org/) - but I've never used it so I can't speak for its performance. Is there a specific reason you're focusing on something like sqlite for in memory database instead of a more traditional system (strictly performance? I don't know your requirements) --- Have you considered SQL Server Compact? (http://www.microsoft.com/en-us/download/details.aspx?id=17876 ) – Nathan May 29 '14 at 21:39
  • OK. Regarding SQL Server Compact - as this isn't an in memory db we left it out of scope. But perhaps I will try it out. Currently the same query performs equally fast on our SQL Server 2014 Standard Edition... :) Our requirements are a fast DB (in memory or not) that interacts good with MS .net and that is widely used and not too pricy. We currently have SQL Server Standard licences so that option is not too bad. – JohanLarsson May 29 '14 at 21:59
  • SQL Server compact creates a file which gets mapped to memory, which is (more or less) the same thing in terms of performance. The benefits of traditional databases are still there, even if it's not purely in memory because of how it can parse out the various fields for indexing/partitioning/etc. But again, I have no knowledge of your project or requirements, so can make only very generalized recommendations such as index, use windows caching if it creates a file, etc. But for numbers: http://stackoverflow.com/questions/5249240/sql-server-ce-4-0-performance-comparison – Nathan May 29 '14 at 22:08
  • Thanks for all your feedback. The query you supplied doesnt seem to be possible with DLinq as I cant find any info how to do it, I also asked another question about this specifically. http://stackoverflow.com/questions/23944024/dynamic-linq-syntax-for-subquery?noredirect=1#comment36882502_23944024 However, I think I have enough info for now. Also I tried the same query in SQL Server Std after having tuned my DB a bit and I am getting better response times in SQL server actually, 0.4 - 0.5 s which might be enough for our purposes – JohanLarsson May 30 '14 at 06:36