How to Write Optimal SQL Queries

Question

I've searched around stackoverflow but everybody asks to optimize queries they've already done.

I want to know, the basic stuff on what to do, what to avoid when creating a query.

For example, It's a known fact that Writing SELECT * FROM is a thing to avoid, given that The sql engine has to make an "invisible" query to know what columns should be shown.

Also know that between @min_number AND @max_number works better than Id >= @min_number AND Id <= @max_number but I don't recall why. It can be because between is a sentence controlled at a lower level by the engine, and creates the iterations to show the regs somehow "handled". But I just don't know for sure.

Could someone validate those and make a list of the most common what to do, what to avoid ?

Answer 1

My list is SQL Server specific (I'm sure that are lots more):

Use sargable where clauses - that means no functions especially scalar UDFs in where clauses among other things

WHERE NOT EXISTS tends to be the faster choice than a left join with a where id is null structure when you are looking for those rows which don't match a second table.

Correlated subqueries tend to run row by row and are horribly slow.

Views that call other views can't be indexed and become very slow especially if you get several levels in on large tables.

Select * is to be avoided espcially when you have a join as at least one column is sent twice which is wasteful of server and database and network resources.

Cursors can usually be replaced with much faster performing set-based logic When you store data in the correct way, you can avoid alot of on-the-fly transformations.

When updating, make sure you add a where clause so that you don't update rows where the new value and the old value are the same. This could be the differnce between updating 10,000,000 rows and updating 15. Sample (Tsql Update structure, if you use another db, you may have to lookup the correct syntax, but it should give you the idea.):

Update t
set field1 = t2.field2
from table1 t
join table2 t2 on t.tid = t2.tid
Where t.field1 <> t2.field2

Or

Update t
set field1 = @variable
from table1 t
Where t.field1 <> @variable

Check your indexing. SQL Seerver does not automatically index foreign keys. If they are used in a join, they generally need to be indexed.

If you are constantly using functions on a field, you are probably not storing it correctly (or you should have a persisted calculated field and do the transformation only once not every time you select the column.)

You best bet is to get a good performance tuning book for your database of choice (what wokrs best is very database specific) and read the chapters concerning writing queries.

Answer 2

• Views are macros, not magic
• EXISTs and NOT EXISTs work best usually
• Functions on columns (see Joel C's answers)
• Beware implicit conversion (eg smallint column compared to int parameter)
• Understand covering indexes
• Denormalise after you see issues
• Understand aggregates: stop thinking of loops
• ...

Edit, Feb 2012:

Avoid these "Ten Common SQL Programming Mistakes"

Answer 3

In your WHERE clause, avoid using a column as an input to a function, as this can cause a full table scan instead of being able to use an index. The query optimizer on some platforms does a better job than others, but it's generally better to be safe. For instance, if you're looking for records from the past 30 days, do the data manipulation against the date you're comparing against, not against your column:

BAD

WHERE DATEADD(DAY, 30, [RecordDate]) > GETDATE()

This may cause a full table scan (depending on the query optimizer for your platform), even if [RecordDate] is indexed, because DATEADD(DAY, 30, [RecordDate]) has to be evaluated to compare it against GETDATE(). If you change it to:

BETTER

WHERE [RecordDate] > DATEADD(DAY, -30, GETDATE())

This will now always be able to use an index on [RecordDate] regardless of how good the query plan optimizer is on your platform, because DATEADD(DAY, -30, GETDATE()) gets evaluated once and can then be used as a lookup in the index. The same principle applies to using a CASE statement, UDF's, etc.

Answer 4

A few general points about optimizing queries:

• Know your data. Know your data. Know your data. I would venture to guess that half of all database performance problems stem from an incomplete understanding of the data and the requirements of the query. Know if your query will be usually returning 50 rows or 5 million rows. Know if you need to get back 3 columns or 50 columns. Know what columns are key columns on the tables, and filter on these.

• Understand your database structure. If you're working with a database in third-normal form, recognize that this structure typically works best on queries for lots of small, transactional statements operating on individual rows. If you are working in a star or snowflake design, recognize that it's optimized for large queries and aggregations.

Answer 5

Here is a good link about Best Practices and performance on SQL server. http://www.sql-server-performance.com/articles/dev/sql_best_practices_p1.aspx

Answer 6

I cant actually validate your claim but can say that not using * sounds quiet logical, what i can do is add a point or two to them, if you can along with giving a select columnname from tablename add a where clause it helps a lot, since you would cut down on a lot of unnecessary rows and rows of data that may be pulled up, also avoiding cross joins and welcoming inner joins, outer joins or fuller joins should be the way to go as per my personal experience :)

Answer 7

My simple rules to write a query:

1. Write FROM clause from the most smallest table. This helps to find data more efficiently as we make searching in smaller amount of data.

2. At first you should write INNER JOIN, then LEFT OUTER JOIN. This helps to decrease quantity of rows where SQL Engine will search your data.

   For example:

   SELECT 
       pe.Name,
       de.Name,
       bu.Name
   FROM dbo.Persons pe
   INNER JOIN dbo.Departments de ON pe.ID = de.id_Person -- at first INNER JOIN
   LEFT JOIN dbo.Bureau bu ON bu.ID = de.id_Bureau -- then LEFT OUTER JOIN
   

3. Use aliases and schema name to avoid schema scanning by SQL Server. As using schema name helps to cashe your query plan for ad-hoc queries that can be reusable by other users, not only for your queries.

4. Avoid using SELECT * ...

Answer 8

From what I've read, using BETWEEN instead of two checks on an index, using AND, improves performance because your database may not fully utilize the benefits of indexes when it sees that it is used on both sides of an AND, or OR.

The query optimizer may not be able to intuit that this is a range check and that the index sorting can come in handy. Instead it may do a scan on each condition and then combine the results. On the other hand, this is very clear with a BETWEEN clause that compares the index column to two values.

Answer 9

adding some tips to the list :

Using EXISTS/NOT EXISTS in place of IN/NOT IN for indexed columns

 --instead of 
 SELECT * FROM table1
  WHERE id1 NOT IN (SELECT id2 FROM table2)

 --you better write
 SELECT * FROM table1 WHERE NOT EXISTS (SELECT 1 FROM table2 WHERE id1=id2)  

Avoid using UNION when its possible to use UNION ALL
when you dont need to exclude duplicated rows or you are sure it wont return duplicated rows

Avoid using HAVING when its possible to use WHERE

 --instead of 
 SELECT col1, sum(col2) 
   FROM table1
  GROUP BY col1
 HAVING col1 > 0

 --you better write :
 SELECT col1, sum(col2)
   FROM table1
  WHERE col1 > 0
 GROUP BY col1

Use EXISTS instead of DISTINCT when you have one-to-many table joins

--instead of
SELECT distinct a.col1, a.col2
  FROM table1 a, table2 b
 WHERE a.id = b.id

--you better write
SELECT a.col1, a.col2
  FROM table1 a
 WHERE EXISTS (SELECT 1 FROM table2 b where a.id = b.id)  

I hope this few tips helps, looking forward more tips ;)