How LINQ works internally?

Question

I love using LINQ in .NET, but I want to know how that works internally?

Answer 1

It makes more sense to ask about a particular aspect of LINQ. It's a bit like asking "How Windows works" otherwise.

The key parts of LINQ are for me, from a C# perspective:

• Expression trees. These are representations of code as data. For instance, an expression tree could represent the notion of "take a string parameter, call the Length property on it, and return the result". The fact that these exist as data rather than as compiled code means that LINQ providers such as LINQ to SQL can analyze them and convert them into SQL.

• Lambda expressions. These are expressions like this:

  x => x * 2
  (int x, int y) => x * y
  () => { Console.WriteLine("Block"); Console.WriteLine("Lambda"); }
  

  Lambda expressions are converted either into delegates or expression trees.

• Anonymous types. These are expressions like this:

  new { X=10, Y=20 }
  

  These are still statically typed, it's just the compiler generates an immutable type for you with properties X and Y. These are usually used with var which allows the type of a local variable to be inferred from its initialization expression.

• Query expressions. These are expressions like this:

  from person in people
  where person.Age < 18
  select person.Name
  

  These are translated by the C# compiler into "normal" C# 3.0 (i.e. a form which doesn't use query expressions). Overload resolution etc is applied afterwards, which is absolutely key to being able to use the same query syntax with multiple data types, without the compiler having any knowledge of types such as Queryable. The above expression would be translated into:

  people.Where(person => person.Age < 18)
        .Select(person => person.Name)
  

• Extension methods. These are static methods which can be used as if they were instance methods of the type of the first parameter. For example, an extension method like this:

  public static int CountAsciiDigits(this string text)
  {
      return text.Count(letter => letter >= '0' && letter <= '9');
  }
  

  can then be used like this:

  string foo = "123abc456";
  int count = foo.CountAsciiDigits();
  

  Note that the implementation of CountAsciiDigits uses another extension method, Enumerable.Count().

That's most of the relevant language aspects. Then there are the implementations of the standard query operators, in LINQ providers such as LINQ to Objects and LINQ to SQL etc. I have a presentation about how it's reasonably simple to implement LINQ to Objects - it's on the "Talks" page of the C# in Depth web site.

The way providers such as LINQ to SQL work is generally via the Queryable class. At their core, they translate expression trees into other query formats, and then construct appropriate objects with the results of executing those out-of-process queries.

Does that cover everything you were interested in? If there's anything in particular you still want to know about, just edit your question and I'll have a go.

Answer 2

LINQ is basically a combination of C# 3.0 discrete features of these:

• local variable type inference
• auto properties (not implemented in VB 9.0)
• extension methods
• lambda expressions
• anonymous type initializers
• query comprehension

For more information about the journey to get there (LINQ), see this video of Anders in LANGNET 2008:

http://download.microsoft.com/download/c/e/5/ce5434ca-4f54-42b1-81ea-7f5a72f3b1dd/1-01%20-%20CSharp3%20-%20Anders%20Hejlsberg.wmv

Answer 3

In simple a form, the compiler takes your code-query and converts it into a bunch of generic classes and calls. Underneath, in case of Linq2Sql, a dynamic SQL query gets constructed and executed using DbCommand, DbDataReader etc.

Say you have:

var q = from x in dc.mytable select x;

it gets converted into following code:

IQueryable<tbl_dir_office> q = 
    dc.mytable.Select<tbl_dir_office, tbl_dir_office>(
        Expression.Lambda<Func<mytable, mytable>>(
            exp = Expression.Parameter(typeof(mytable), "x"), 
            new ParameterExpression[] { exp }
        )
    );

Lots of generics, huge overhead.

Answer 4

Basically linq is a mixture of some language facilities (compiler) and some framework extensions. So when you write linq queries, they get executed using appropriate interfaces such as IQuerable. Also note that the runtime has no role in linq.

But it is difficult to do justice to linq in a short answer. I recommend you read some book to get yourself in it. I am not sure about the book that tells you internals of Linq but Linq in Action gives a good handson about it.