Modify the expression tree of IQueryable.Include() to add condition to the join

Question

Basically, I would like to implement a repository that filters all the soft deleted records even through navigation properties. So I have a base entity, something like that:

public abstract class Entity
{
    public int Id { get; set; }

    public bool IsDeleted { get; set; }

    ...
}

And a repository:

public class BaseStore<TEntity> : IStore<TEntity> where TEntity : Entity
{
    protected readonly ApplicationDbContext db;

    public IQueryable<TEntity> GetAll()
    {
        return db.Set<TEntity>().Where(e => !e.IsDeleted)
            .InterceptWith(new InjectConditionVisitor<Entity>(entity => !entity.IsDeleted));
    }

    public IQueryable<TEntity> GetAll(Expression<Func<TEntity, bool>> predicate)
    {
        return GetAll().Where(predicate);
    }

    public IQueryable<TEntity> GetAllWithDeleted()
    {
        return db.Set<TEntity>();
    }

    ...
}

The InterceptWith function is from this projects: https://github.com/davidfowl/QueryInterceptor and https://github.com/StefH/QueryInterceptor (same with async implementations)

A usage of an IStore<Project> looks like:

var project = await ProjectStore.GetAll()
          .Include(p => p.Versions).SingleOrDefaultAsync(p => p.Id == projectId);

I implemented an ExpressionVisitor:

internal class InjectConditionVisitor<T> : ExpressionVisitor
{
    private Expression<Func<T, bool>> queryCondition;

    public InjectConditionVisitor(Expression<Func<T, bool>> condition)
    {
        queryCondition = condition;
    }

    public override Expression Visit(Expression node)
    {
        return base.Visit(node);
    }
}

But this is the point where I got stucked. I put a breakpoint in the Visit function to see what expressions I got, and when should I do somthing cheeky, but it never gets to the Include(p => p.Versions) part of my tree.

I saw some other solutions that may work, but those are "permanent", for example EntityFramework.Filters seemed to be good for the most use-cases, but you have to add a filter when you are configuring the DbContext - however, you can disable filters, but I do not want to disable and reenable a filter for every query. Another solution like this is to subscribe the ObjectContext's ObjectMaterialized event but I would not like it either.

My goal would be to "catch" the includes in the visitor and modify the expression tree to add another condition to the join which checks the IsDeleted field of the record only if you use one of the GetAll function of the store. Any help would be appreciated!

Update

The purpose of my repositories is to hide some basic behavior of the base Entity - it also contains "created/lastmodified by", "created/lastmodified-date", timestamp, etc. My BLL gets all the data through this repositories so it does not need to worry about those, the store will handle all the things. There is also a possibility to inherit from the BaseStore for a specific class (then my configured DI will inject to inherited class into IStore<Project> if it exists), where you can add specific behavior. For example if you modify a project, you need to add these modification historical, then you just add this to the update function of the inherited store.

The problem starts when you querying a class that has navigation properties (so any class :D ). There is two concrete entity:

  public class Project : Entity 
  {
      public string Name { get; set; }

      public string Description { get; set; }

      public virtual ICollection<Platform> Platforms { get; set; }

      //note: this version is not historical data, just the versions of the project, like: 1.0.0, 1.4.2, 2.1.0, etc.
      public virtual ICollection<ProjectVersion> Versions { get; set; }
  }

  public class Platform : Entity 
  {
      public string Name { get; set; }

      public virtual ICollection<Project> Projects { get; set; }

      public virtual ICollection<TestFunction> TestFunctions { get; set; }
  }

  public class ProjectVersion : Entity 
  {
      public string Code { get; set; }

      public virtual Project Project { get; set; }
  }

So if I would like to list the versions of the project, I call the store: await ProjectStore.GetAll().Include(p => p.Versions).SingleOrDefaultAsync(p => p.Id == projectId). I will not get deleted project, but if the project exists, it will give back all the Versions related to it, even the deleted ones. In this specific case, I could start from the other side and call the ProjectVersionStore, but if I would like to query through 2+ navigation properties then it's game end:)

The expected behavior would be: if I include the Versions to the Project, it should query only the not deleted Versions - so the generated sql join should contains a [Versions].[IsDeleted] = FALSE condition also. It is even more complicated with complex includes like Include(project => project.Platforms.Select(platform => platform.TestFunctions)).

The reason I'm trying to do it this way is I don't want to refactor all the Include's in the BLL to something else. That's the lazy part:) The another is I would like a transparent solution, I don't want the BLL to know all of this. The interface should be kept unchanged if it is not absolutely necessary. I know it's just an extension method, but this behavior should be in the store layer.

Answer 1

The include method you use calls the method QueryableExtensions.Include(source, path1) which transforms the expression into a string path. This is what the include method does:

public static IQueryable<T> Include<T, TProperty>(this IQueryable<T> source, Expression<Func<T, TProperty>> path)
{
  Check.NotNull<IQueryable<T>>(source, "source");
  Check.NotNull<Expression<Func<T, TProperty>>>(path, "path");
  string path1;
  if (!DbHelpers.TryParsePath(path.Body, out path1) || path1 == null)
    throw new ArgumentException(Strings.DbExtensions_InvalidIncludePathExpression, "path");
  return QueryableExtensions.Include<T>(source, path1);
}

So, your expression looks like this (check the "Include" or "IncludeSpan" method in your expression):

 value(System.Data.Entity.Core.Objects.ObjectQuery`1[TEntity]).MergeAs(AppendOnly)
   .IncludeSpan(value(System.Data.Entity.Core.Objects.Span))

You should hook on VisitMethodCall to add your expression instead:

internal class InjectConditionVisitor<T> : ExpressionVisitor
{
    private Expression<Func<T, bool>> queryCondition;

    protected override Expression VisitMethodCall(MethodCallExpression node)
    {
        Expression expression = node;
        if (node.Method.Name == "Include" || node.Method.Name == "IncludeSpan")
        {
            // DO something here! Let just add an OrderBy for fun

            // LAMBDA: x => x.[PropertyName]
            var parameter = Expression.Parameter(typeof(T), "x");
            Expression property = Expression.Property(parameter, "ColumnInt");
            var lambda = Expression.Lambda(property, parameter);

            // EXPRESSION: expression.[OrderMethod](x => x.[PropertyName])
            var orderByMethod = typeof(Queryable).GetMethods().First(x => x.Name == "OrderBy" && x.GetParameters().Length == 2);
            var orderByMethodGeneric = orderByMethod.MakeGenericMethod(typeof(T), property.Type);
            expression = Expression.Call(null, orderByMethodGeneric, new[] { expression, Expression.Quote(lambda) });
        }
        else
        {
            expression = base.VisitMethodCall(node);
        }

        return expression;
    }
}

The QueryInterceptor project from David Fowl doesn't support "Include". Entity Framework tries to find the "Include" method using reflection and return the current query if not found (which is the case).

Disclaimer: I'm the owner of the project EF+.

I have added a QueryInterceptor feature which supports "Include" to answer your question. The feature is not yet available since unit test has not been added but you can download and try the source: Query Interceptor Source

Contact me directly (email at the bottom of my GitHub homepage) if you have an issue, since this will start to be off topic otherwise.

Be careful, "Include" method modifies the expression by hiding some previous expressions. So it's sometime hard to understand what's really happening under the hood.

My project also contains a Query Filter feature which I believe has more flexibility.

---

EDIT: Add working example from updated required

Here is a starting code you can use for your requirement:

public IQueryable<TEntity> GetAll()
{
    var conditionVisitor = new InjectConditionVisitor<TEntity>("Versions", db.Set<TEntity>.Provider, x => x.Where(y => !y.IsDeleted));
    return db.Set<TEntity>().Where(e => !e.IsDeleted).InterceptWith(conditionVisitor);
}

var project = await ProjectStore.GetAll().Include(p => p.Versions).SingleOrDefaultAsync(p => p.Id == projectId);

internal class InjectConditionVisitor<T> : ExpressionVisitor
{
    private readonly string NavigationString;
    private readonly IQueryProvider Provider;
    private readonly Func<IQueryable<T>, IQueryable<T>> QueryCondition;

    public InjectConditionVisitor(string navigationString, IQueryProvider provder , Func<IQueryable<T>, IQueryable<T>> queryCondition)
    {
        NavigationString = navigationString;
        Provider = provder;
        QueryCondition = queryCondition;
    }

    protected override Expression VisitMethodCall(MethodCallExpression node)
    {
        Expression expression = node;

        bool isIncludeSpanValid = false;

        if (node.Method.Name == "IncludeSpan")
        {
            var spanValue = (node.Arguments[0] as ConstantExpression).Value;

            // The System.Data.Entity.Core.Objects.Span class and SpanList is internal, let play with reflection!
            var spanListProperty = spanValue.GetType().GetProperty("SpanList", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
            var spanList = (IEnumerable)spanListProperty.GetValue(spanValue);

            foreach (var span in spanList)
            {
                var spanNavigationsField = span.GetType().GetField("Navigations", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
                var spanNavigation = (List<string>)spanNavigationsField.GetValue(span);

                if (spanNavigation.Contains(NavigationString))
                {
                    isIncludeSpanValid = true;
                    break;
                }
            }
        }

        if ((node.Method.Name == "Include" && (node.Arguments[0] as ConstantExpression).Value.ToString() == NavigationString)
            || isIncludeSpanValid)
        {

            // CREATE a query from current expression
            var query = Provider.CreateQuery<T>(expression);

            // APPLY the query condition
            query = QueryCondition(query);

            // CHANGE the query expression
            expression = query.Expression;
        }
        else
        {
            expression = base.VisitMethodCall(node);
        }

        return expression;
    }
}

---

EDIT: Answer sub questions

Difference between Include and IncludeSpan

From what I understand

IncludeSpan: Appears when the original query has not been yet modified by a LINQ method.

Include: Appears when the original query has been modified by a LINQ method (You do not longer see previous expression)

-- Expression: {value(System.Data.Entity.Core.Objects.ObjectQuery`1[Z.Test.EntityFramework.Plus.Association_Multi_OneToMany_Left]).MergeAs(AppendOnly).IncludeSpan(value(System.Data.Entity.Core.Objects.Span))}
var q = ctx.Association_Multi_OneToMany_Lefts.Include(x => x.Right1s).Include(x => x.Right2s);


-- Expression: {value(System.Data.Entity.Core.Objects.ObjectQuery`1[Z.Test.EntityFramework.Plus.Association_Multi_OneToMany_Left]).Include("Right2s")}
var q = ctx.Association_Multi_OneToMany_Lefts.Include(x => x.Right1s).Where(x => x.ColumnInt > 10).Include(x => x.Right2s);

How to include and filter related entities

Include does not let you filter related entities. You can find 2 solutions in this post: EF. How to include only some sub results in a model?

• One involve using a projection
• One involve using EF+ Query IncludeFilter from my library