1

I have designed a recursive SQL CTE expression that sorts a recordset according to its parent Id in a nested hierarchy. How can I execute use this CTE query in my EF6 data context?

  • I was expecting to find a way to define CTEs in linq statements.

For background, this previous post helped me to identify the CTE: Order By SQL Query based on Value another Column.

For the purposes of this post I am using a single table in the EF context. This data model class has been generated from the database using Entity Framework 6 (ADO.NET Entity Data Model)

public partial class Something
{
    public int Id{ get; set; }
    public string Name{ get; set; }
    public string Address { get; set; }
    public string Email { get; set; }
    public string PhoneNumber { get; set; }
    public System.DateTime Date { get; set; }
    public int IdParent{ get; set; }
}

And this is the sql query that I want to execute or translate to Linq

with cte (Id, Name, Address, Email, PhoneNumber, Date, IdParent, sort) as
(
  select Id, Name, Address, Email, PhoneNumber, Date, IdParent,
         cast(right('0000' + cast(row_number() over (order by Id) as varchar(5)), 5) as varchar(1024))
  from   Something
  where  Id = IdParent 

  union all

  select t.Id, t.Name, t.Address, t.Email, t.PhoneNumber, t.Date, t.IdParent,
         cast(c.sort + right('0000' + cast(row_number() over (order by t.Id) as varchar(5)), 5) as varchar(1024))
  from   cte c                                   
         inner join Something t on c.Id = t.IdParent 
  where  t.Id <> t.IdParent 
)
select *
from cte
order by sort
Chris Schaller
  • 13,704
  • 3
  • 43
  • 81

4 Answers4

3

Writing hierachial queries in Linq to SQL is always a mess, it can work in memory but it doesn't translate to efficient SQL queries, this is a good discussion on on SO about some hierarchial Linq techniques

There are a few options:

  1. Don't use Linq at all and query from your CTE directly!
  2. Convert your CTE to a View
  3. Re-write the query so that you don't need the CTE
    • This is easier if you have a fixed or theoretical limit to the recursion.
    • Even if you don't want to limit it, if you review the data and find that the highest level of recursion is only 2 or 3, then you could support

How to use a CTE directly in EF 6

DbContext.Database.SqlQuery<TElement>(string sql, params object[] parameters)
Creates a raw SQL query that will return elements of the given type (TElement). The type can be any type that has properties that match the names of the columns returned from the query

NOTE: Do NOT use select * for this type (or any) of query, explicitly define the fields that you expect in the output to avoid issues where your query has more columns available than the EF runtime is expecting.

  • Perhaps of equal importance, if you want or need to apply filtering to this record set, you should implement the filtering in the raw SQL string value. The entire query must be materialized into memory before EF Linq filtering expressions can be applied
    • .SqlQuery does support passing through parameters, which comes in handy for filter expressions ;)
string cteQuery = @"
with cte (Id, Name, Address, Email, PhoneNumber, Date, IdParent, sort) as
(
  select Id, Name, Address, Email, PhoneNumber, Date, IdParent,
         cast(right('0000' + cast(row_number() over (order by Id) as varchar(5)), 5) as varchar(1024))
  from   Something
  where  Id = IdParent 

  union all

  select t.Id, t.Name, t.Address, t.Email, t.PhoneNumber, t.Date, t.IdParent,
         cast(c.sort + right('0000' + cast(row_number() over (order by t.Id) as varchar(5)), 5) as varchar(1024))
  from   cte c                                   
         inner join Something t on c.Id = t.IdParent 
  where  t.Id <> t.IdParent 
)
select Id, Name, Address, Email, PhoneNumber, Date, IdParent
from cte
order by sort
";

using (var ctx = new MyDBEntities())
{
    var list = ctx.Database
                  .SqlQuery<Something>(cteQuery)
                  .ToList();
}
  • Understanding how and when to use .SqlQuery for executing raw SQL comes in handy when you want to squeeze the most performance out of SQL without writing complex Linq statements.
  • This comes in handy if you move your CTE into a view or table valued function or a stored procedure, once the results have been materialized into the list in memory, you can treat these records like any other

Convert your CTE to a View

If you are generating your EF model from the database, then you could create a view from your CTE to generate the Something class, however this becomes a bit disconnected if you also want to perform CRUD operations against the same table, having two classes in the model that represent virtually the same structure is a bit redundant IMO, perfectly valid if you want to work that way though.

  • Views cannot have ORDER BY statements, so you take this statement out of your view definition, but you still include the sort column in the output so that you can sort the results in memory.
  • Converting your CTE to a view will have the same structure as your current Something class, however it will have an additional column called sort.

How to write the same query without CTE

As I alluded at the start, you can follow this post Hierarchical queries in LINQ to help process the data after bringing the entire list into memory. However in my answer to OPs orginal post, I highlighted how simple self joins on the table can be used to produce the same results, we can easily replicate the self join in EF.

Even when you want to support a theoretically infinitely recursive hierarchy the realty of many datasets is that there is an observable or practical limit to the number of levels. If you can identify that practical limit, and it is a small enough number, then it might be simpler from a C# / Linq perspective to mot bother with the CTE at all

  • Put it the other way around, ask yourself this question: "If I set a practical limit of X number of levels of recursion, how will that affect my users?"
    • Put 4 in for X, if the result is that users will not generally be affected, or this data scenario is not likely to occur then lets try it out.

If a limit of 4 is acceptable, then this is your Linq statement:
I've used fluent notation here to demonstrate the relationship to SQL

var list = from child in ctx.Somethings
           join parent in ctx.Somethings on child.parentId equals parent.Id
           join grandParent in ctx.Somethings on parent.parentId equals grandParent.Id
           orderby grandParent.parentId, parent.parentId, child.parentId, child.Id
           select child;

I would probably use short hand aliases for this query in production, but the naming convention makes the intended query quickly human relatable.

If you setup a foreign key in the database linking parentId to the Id of the same table, then the Linq side is much simpler

  • This should generate a navigation property to enable traversing the foreign key through linq, in the following example this property is called Parent
var list = ctx.Somethings
              .OrderBy(x => x.Parent.Parent.ParentId)
              .ThenBy(x => x.Parent.ParentId)
              .ThenBy(x => x.ParentId)
              .ThenBy(x => x.Id);

You can see in this way, if we can limit the recusion level, the Linq required for sorting based on the recursive parent is quite simple, and syntactically easy to extrapolate to the number of levels you need.

  • You could do this for any number of levels, but there is a point where performance might become an issue, or where the number of line of code to achieve this is more than using the SqlQuery option presented first.
Chris Schaller
  • 13,704
  • 3
  • 43
  • 81
0

I'd recommend you create a view using the SQL provided. Once you create a view, you can map the view to a C# DTO using Entity Framework. After that you can query the view using LINQ.

Don't forget to include the [sort] column in your DTO because you can't include (or at least shouldn't) the sort order in your view definition. You can sort the query using LINQ instead of SQL directly.

Jason
  • 711
  • 6
  • 14
0

LINQ for SQL CTE

Here is an approach that applies to many scenarios where LINQ shouldn't/can't/won't/will never produce the SQL you need. This example executes a raw SQL CTE driven by LINQ logic to return the primary key (PK) value's ordinal row number regardless of basic sorts and filters on the entity/table.

The goal is to apply the same constraints to differing requirements. One requirement is the PK row's position w/in those constraints. Another requirement may be the count of rows that satisfy those constraints, etc. Those statistics need to be based on a common constraint broker.

Here, an IQueryable, under the purview of an open DbContext, applies those constraints and is the constraint broker. An alternate approach outside of any DbContext purview is to build expression trees as the constraint broker and return them for evaluation once back under the DbContext umbrella. The shortcut to that is https://github.com/dbelmont/ExpressionBuilder

LINQ could not express Structured Query Language (SQL) Common Table Expressions (CTE) in previous .NET versions. LINQ still can't do that. But...

Hello .NET 5 and my new girlfriend, IQueryable.ToQueryString(). She's the beautiful but potentially lethal kind. Regardless, she gives me all the target row numbers I could ever want.

But, I digress...

/// <summary>
/// Get the ordinal row number of a given primary key value within filter and sort constraints
/// </summary>
/// <param name="TargetCustomerId">The PK value to find across a sorted and filtered record set</param>
/// <returns>The ordinal row number (where the 1st filtered & sorted row is #1 - NOT zero), amongst all other filtered and sorted rows, for further processing - like conversion to page number per a rows-per-page value</returns>
/// <remarks>Doesn't really support fancy ORDER BY clauses here</remarks>
public virtual async Task<int> GetRowNumber(int TargetCustomerId)
    {
        int rowNumber = -1;
    
        using (MyDbContext context = new MyDbContext())
        {
            // Always require a record order for row number CTEs
            string orderBy = "LastName, FirstName";
    
            // Create a query with a simplistic SELECT but all required Where() criteria
            IQueryable<Customer> qrbl = context.Customer
                // .Includes are not necessary for filtered row count or the row number CTE
                .Include(c => c.SalesTerritory)
                    .ThenInclude(sr => sr.SalesRegion)
                .Where(c => c.AnnualIncome > 30000 && c.SalesTerritory.SalesRegion.SalesRegionName == "South")
                .Select(c => c )
                ;
    
            // The query doesn't necessarily need to be executed...
            // ...but for pagination, the filtered row count is valuable for UI stuff - like a "page x of n" pagination control, accurate row sliders or scroll bars, etc.
            // int custCount = Convert.ToInt32(await qrbl.CountAsync());
    
            // Extract LINQ's rendered SQL
            string sqlCustomer = qrbl.ToQueryString();
    
            // Remove the 1st/outer "SELECT" clause from that extracted sql
            int posFrom = sqlCustomer.IndexOf("FROM [schemaname].[Customer] ");
            string cteFrom = sqlCustomer.Substring(posFrom);
    
            /*
                If you must get a row number from a more complex query, where LINQ nests SELECTs, this approach might be more appropriate.
                string[] clauses = sqlCustomer.Split("\r\n", StringSplitOptions.TrimEntries);
                int posFrom = clauses
                    .Select((clause, index) => new { Clause = clause, Index = index })
                    .First(ci => ci.Clause.StartsWith("FROM "))
                    .Index
                    ;
                string cteFrom = string.Join("\r\n", clauses, posFrom, clauses.Length - posFrom);
            */
    
            // As always w/ all raw sql, prohibit sql injection, etc.
            string sqlCte = "WITH cte AS\r\n"
                + $"\t(SELECT [CustomerId], ROW_NUMBER() OVER(ORDER BY {orderBy}) AS RowNumber {cteFrom})\r\n"
                + $"SELECT @RowNumber = RowNumber FROM cte WHERE [CustomerId] = {TargetCustomerId}"
                ;
            SqlParameter paramRow = new SqlParameter("RowNumber", System.Data.SqlDbType.Int);
            paramRow.Direction = System.Data.ParameterDirection.Output;
            int rows = await context.Database.ExecuteSqlRawAsync(sqlCte, paramRow).ConfigureAwait(false);
    
            if (paramRow.Value != null)
            {
                rowNumber = (int)paramRow.Value;
            }
        }
    
        return rowNumber;
    }
marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
KramFfud
  • 179
  • 1
  • 2
  • 6
0

Use your CTE With A different ORM

One can call stored procedures from EF and there are a number of posts to that end which you can do... but I recommend that you do a hybrid EF and ADO.Net system which will take advantage of your specialized sql code.

ADO.Net can be used which you will have to write by hand... but there is an ado.net based ORM which uses the principle of returning JSON from SQL Server as models. This ORM can be installed side by side with EF.

It is the Nuget Package SQL-Json (which I am the author) which can use your CTE and provide data as an array of models for your code to use.


Steps

  1. Have your final CTE output return a JSON data by adding for json auto;.
  2. Run the sql and generate json. Take that json to create a C# model using any website which coverts JSON To C# classes. In this example let us call the model class CTEData.
  3. Put your sql into a store procedure.
  4. Include SQL-Json package into your project.
  5. In the model created in step #2 inherit the base class JsonOrmModel.
  6. In the model again add this override public override string GetStoredProcedureName => "[dbo].MyGreatCTE"; with your actual sproc created in step #3.
  7. Get the models:
var connectionStr = @"Data Source=.\Jabberwocky;Initial Catalog=WideWorldImporters";

var jdb = new JsonOrmDatabase(connectionStr);
    
List<CTEData> ctes = jdb.Get<CTEData>();
    

Then you can use your cte data as needed.


On the project page it shows how to do what I described with a basic POCO model at SQL-Json.

ΩmegaMan
  • 29,542
  • 12
  • 100
  • 122