12

I don't understand why this doesn't translate. It seems to be exactly the use case described here.

The LINQ expression

DbSet<A>()
    .GroupJoin(
        inner: DbSet<B>(),
        outerKeySelector: a => a.AId,
        innerKeySelector: b => b.AId,
        resultSelector: (a, bs) => new {
            a = a,
            bs = bs
         })

produces the error:

could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.

The LINQ code producing the exception is

from a in ctx.As
    join b in ctx.Bs on a.aId equals b.aId into bs
    select new {A = a, Bs = bs.ToList()};

Edit: maybe I misunderstood the doc and this is an example of something that does NOT translate.

Executing a query like the following example generates a result of Blog & IEnumerable. Since databases (especially relational databases) don't have a way to represent a collection of client-side objects, GroupJoin doesn't translate to the server in many cases. It requires you to get all of the data from the server to do GroupJoin without a special selector (first query below). But if the selector is limiting data being selected then fetching all of the data from the server may cause performance issues (second query below). That's why EF Core doesn't translate GroupJoin.

But then my question becomes instead : how do I achieve the result I'm looking for without requiring navigational properties ?

Ian Kemp
  • 28,293
  • 19
  • 112
  • 138
Julien Debache
  • 319
  • 5
  • 13
  • The last sentence of the quoted documentation is the giveaway here: _"That's why EF Core doesn't translate GroupJoin."_ – phuzi Aug 31 '23 at 23:12

4 Answers4

21

This functionality has been implemented as of EF Core 7. Not exactly what we wanted (leads to OUTER APPLY in SQL rather than LEFT JOIN as with manual correlated subquery shown below, or collection navigation property), but at least is translated.


The explanation in the linked documentation just follows the EF Core team vision and is ridiculous, because of course it can easily be translated - I had a long discussion with the team here Query with GroupBy or GroupJoin throws exception #17068 and continue here Query: Support GroupJoin when it is final query operator #19930, trying to convince them why it should be supported, with no luck regardless of the arguments.

The whole point is (and that's the current workaround) it can be processed like if it was correlated subquery (SelectMany), which is translated and processed properly (even though the query result shape has no SQL equivalent.

Anyway, the current status is "Needs Design" (whatever that means), and the workaround is to replace the join with correlated subquery (which is what EF Core is using internally when "expanding" collection navigation properties during the query translation).

In your case, replace

join b in ctx.Bs on a.aId equals b.aId into bs

with

let bs = ctx.Bs.Where(b => a.aId == b.aId)

However, I highly recommend adding and using navigation properties. Not sure why you "can't use" them, in LINQ to Entities which do not project entities they serve just metadata for relationships, thus produce automatically the necessary joins. By not defining them you just put on yourself unneeded limitations (additionally to EF Core limitations/bugs). In general EF Core works better and support more things when using navigation properties instead of manual joins.

Ivan Stoev
  • 195,425
  • 15
  • 312
  • 343
  • Thanks, this seems to be what I'm looking for. I don't use navigational properties because they're error prone in my use cases at least. Also, I prefer making the joins explicit and having simpler entity types s.t. I don't forget about the limitations of relation database management – Julien Debache Mar 28 '21 at 20:33
  • 4
    Hi Ivan, Keep up the good fight. It seems very strange that the translation of GroupJoin worked perfectly in EF6 but when someone switch to EF core then workaround has to be used. – Istvan Heckl Dec 29 '21 at 11:15
  • Navigation property cannot be used for arbitrary joins unfortunately. It requires an established foreign-principal key relationship and if the principal key is a composite, it cannot be null. A regular join works in that situation just fine and a group join should too. – Kristaps Baumanis Apr 20 '23 at 12:25
7

EF Core will not allow you to do a GroupJoin without following up with a SelectMany in order to flatten the list. The GroupJoin has no equivalent implementation in SQL, however the GroupJoin/SelectMany is equivalent to an inner-join or left-join (depending on if you use DefaultIfEmpty) so it works fine:

context.Users.GroupJoin(
        context.UserRoles,
        u => u.UserId,
        r => r.UserId,
        (user, roles) => new { user, roles })

    //Will not work without this line
    .SelectMany(x => x.roles.DefaultIfEmpty(), (x, r) => new { x.user, role = r })
    .ToList();

If you actually want your results to be grouped (as opposed to trying to do a left-join) you have a few options:

  1. You can materialize the results of a left join, then group the results in-memory (the code below uses my LeftJoin function shown in LEFT OUTER JOIN in LINQ):

    context.Users.LeftJoin(
                context.UserRoles,
                u => u.UserId,
                r => r.UserId,
                (user, roles) => new { user, roles })
            .ToList()
            .GroupBy(x => x.user, (u, x) => new
            {
                User = u,
                Roles = x.Select(z => z.role).Where(r => r != null).ToList()
            })
            .ToList();
    
  2. You can use a sub-query. Note that EF is smart enough to use a left-join when it generates the SQL:

    context.Users.Select(u => new
        {
            User = u,
            Roles = context.UserRoles.Where(r => r.UserId == u.UserId).ToList()
        })
        .ToList();
    

If you prefer the GroupJoin syntax, but don't want to have to keep calling all the other functions to flatten, materialize, then re-group the results, you can use my JoinMany() extension method. This method uses the sub-query approach but wraps it in a generic method that looks very similar to the GroupJoin function:

    context.Users.JoinMany(
            context.UserRoles,
            (u, r) => u.UserId == r.UserId,
            (user, roles) => new { user, roles })
        .ToList();

Supporting code:

public static class QueryableExtensions
{
    public static IQueryable<TResult> LeftJoin<TOuter, TInner, TKey, TResult>(
           this IQueryable<TOuter> outer,
           IEnumerable<TInner> inner, Expression<Func<TOuter, TKey>> outerKeySelector,
           Expression<Func<TInner, TKey>> innerKeySelector,
           Expression<Func<TOuter, TInner, TResult>> resultSelector)
    {
        return outer
            .GroupJoin(inner, outerKeySelector, innerKeySelector, (o, i) => new { o, i })
            .SelectMany(o => o.i.DefaultIfEmpty(), (x, i) => new { x.o, i })
            .ApplySelector(x => x.o, x => x.i, resultSelector);
    }

    public static IQueryable<TResult> JoinMany<TOuter, TInner, TResult>(
        this IQueryable<TOuter> outers, IQueryable<TInner> inners,
        Expression<Func<TOuter, TInner, bool>> condition,
        Expression<Func<TOuter, IEnumerable<TInner>, TResult>> resultSelector)
    {
        //Use a placeholder "p => true" expression for the sub-query
        Expression<Func<TOuter, JoinResult<TOuter, IEnumerable<TInner>>>> joinSelector = o =>
            new JoinResult<TOuter, IEnumerable<TInner>> { Outer = o, Inner = inners.Where(p => true) };

        //Create the where-clause that will be used for the sub-query
        var whereClause = Expression.Lambda<Func<TInner, bool>>(
                        condition.Body.ReplaceParameter(condition.Parameters[0], joinSelector.Parameters[0]),
                        condition.Parameters[1]);

        //Replace the placeholder expression with our new where clause
        joinSelector = Expression.Lambda<Func<TOuter, JoinResult<TOuter, IEnumerable<TInner>>>>(
            joinSelector.Body.VisitExpression(node =>
                (node is LambdaExpression le && le.Parameters.Count == 1 && le.Parameters[0].Type == typeof(TInner)
                    && le.Body is ConstantExpression ce && ce.Value is bool b && b)
                    ? whereClause : null),
            joinSelector.Parameters[0]);

        return outers.Select(joinSelector).ApplySelector(x => x.Outer, x => x.Inner, resultSelector);
    }

    private static IQueryable<TResult> ApplySelector<TSource, TOuter, TInner, TResult>(
        this IQueryable<TSource> source,
        Expression<Func<TSource, TOuter>> outerProperty,
        Expression<Func<TSource, TInner>> innerProperty,
        Expression<Func<TOuter, TInner, TResult>> resultSelector)
    {
        var p = Expression.Parameter(typeof(TSource), $"param_{Guid.NewGuid()}".Replace("-", string.Empty));
        Expression body = resultSelector?.Body
            .ReplaceParameter(resultSelector.Parameters[0], outerProperty.Body.ReplaceParameter(outerProperty.Parameters[0], p))
            .ReplaceParameter(resultSelector.Parameters[1], innerProperty.Body.ReplaceParameter(innerProperty.Parameters[0], p));
        var selector = Expression.Lambda<Func<TSource, TResult>>(body, p);
        return source.Select(selector);
    }

    public class JoinResult<TOuter, TInner>
    {
        public TOuter Outer { get; set; }
        public TInner Inner { get; set; }
    }
}


public static class ExpressionExtensions
{
    public static Expression ReplaceParameter(this Expression source, ParameterExpression toReplace, Expression newExpression)
        => new ReplaceParameterExpressionVisitor(toReplace, newExpression).Visit(source);

    public static Expression VisitExpression(this Expression source, Func<Expression, Expression> onVisit)
        => new DelegateExpressionVisitor (onVisit).Visit(source);
}

public class DelegateExpressionVisitor : ExpressionVisitor
{
    Func<Expression, Expression> OnVisit { get; }

    public DelegateExpressionVisitor(Func<Expression, Expression> onVisit)
    {
        this.OnVisit = onVisit;
    }

    public override Expression Visit(Expression node)
    {
        return OnVisit(node) ?? base.Visit(node);
    }
}

public class ReplaceParameterExpressionVisitor : ExpressionVisitor
{
    public ParameterExpression ToReplace { get; }
    public Expression ReplacementExpression { get; }

    public ReplaceParameterExpressionVisitor(ParameterExpression toReplace, Expression replacement)
    {
        this.ToReplace = toReplace;
        this.ReplacementExpression = replacement;
    }

    protected override Expression VisitParameter(ParameterExpression node)
        => (node == ToReplace) ? ReplacementExpression : base.VisitParameter(node);
}
NuclearProgrammer
  • 806
  • 1
  • 9
  • 19
3

Try this query, it should work with EF Core:

var query =
    from a in ctx.As
    select new {A = a, Bs = ctx.Bs.Where(b => b.Id == a.aId).ToList()};
Svyatoslav Danyliv
  • 21,911
  • 3
  • 16
  • 32
0

Probably the .ToList() cannot be translated. Use include instead

var result = ctx.As
    .Include(a => a.Bs)
    .ToList();

Where you must have a navigation property for the Bs in the A class:

public class A
{
    public int aId { get; set; }
    public List<B> Bs { get; set; }
}

See:

Olivier Jacot-Descombes
  • 104,806
  • 13
  • 138
  • 188
  • Sadly, I cannot use navigational properties. Let me try without the `ToList()` though. Edit : removing the ToList() does not solve the issue. – Julien Debache Mar 28 '21 at 19:08