How to write left outer join using MethodCallExpressions?

Question

The code block below answers the question: "How do you perform a left outer join using linq extension methods?"

var qry = Foo.GroupJoin(
      Bar, 
      foo => foo.Foo_Id,
      bar => bar.Foo_Id,
      (x,y) => new { Foo = x, Bars = y })
.SelectMany(
      x => x.Bars.DefaultIfEmpty(),
      (x,y) => new { Foo = x, Bar = y});

How do you write this GroupJoin and SelectMany as MethodCallExpressions? All of the examples that I've found are written using DynamicExpressions translating strings into lambdas (another example). I like to avoid taking a dependency on that library if possible.

Can the query above be written with Expressions and associated methods?

I know how to construct basic lambda expressions like foo => foo.Foo_Id using ParameterExpressions MemberExpressions and Expression.Lambda() , but how do you construct (x,y) => new { Foo = x, Bars = y })??? to be able to construct the necessary parameters to create both calls?

MethodCallExpression groupJoinCall =
         Expression.Call(
           typeof(Queryable),
           "GroupJoin",
           new Type[] { 
                  typeof(Customers), 
                  typeof(Purchases), 
                  outerSelectorLambda.Body.Type, 
                  resultsSelectorLambda.Body.Type 
               },
                           c.Expression,
                           p.Expression,
                           Expression.Quote(outerSelectorLambda),
                           Expression.Quote(innerSelectorLambda),
                           Expression.Quote(resultsSelectorLambda)
                        );
MethodCallExpression selectManyCall =
   Expression.Call(typeof(Queryable), 
    "SelectMany", new Type[] { 
        groupJoinCall.ElementType, 
        resultType, 
        resultsSelectorLambda.Body.Type 
    }, groupJoinCall.Expression, Expression.Quote(lambda), 
    Expression.Quote(resultsSelectorLambda))); 

Ultimately, I need to create a repeatable process that will left join n Bars to Foo. Because we have a vertical data structure, a left-joined query is required to return what is represented as Bars, to allow the user to sort Foo. The requirement is to allow the user to sort by 10 Bars, but I don't expect them to ever use more than three. I tried writing a process that chained the code in the first block above up to 10 times, but once I got passed 5 Visual Studio 2012 start to slow and around 7 it locked up.

Therefore, I'm now trying to write a method that returns the selectManyCall and calls itself recursively as many times as is requested by the user.

Based upon the query below that works in LinqPad, the process that needs to be repeated only requires manually handling the transparent identifiers in Expression objects. The query sorts returns Foos sorted by Bars (3 Bars in this case).

A side note. This process is significantly easier doing the join in the OrderBy delegate, however, the query it produces includes the T-SQL "OUTER APPLY", which isn't supported by Oracle which is required.

I'm grateful for any ideas on how to write the projection to anonymous type or any other out-of-the-box idea that may work. Thank you.

var q = Foos
        .GroupJoin (
            Bars, 
            g => g.FooID, 
            sv => sv.FooID, 
            (g, v) => 
                new  
                {
                    g = g, 
                    v = v
                }
        )
        .SelectMany (
            s => s.v.DefaultIfEmpty (), 
            (s, v) => 
                new  
                {
                    s = s, 
                    v = v
                }
        )
        .GroupJoin (
            Bars, 
            g => g.s.g.FooID, 
            sv => sv.FooID, 
            (g, v) => 
                new  
                {
                    g = g, 
                    v = v
                }
        )
        .SelectMany (
            s => s.v.DefaultIfEmpty (), 
            (s, v) => 
                new  
                {
                    s = s, 
                    v = v
                }
        )
        .GroupJoin (
            Bars,  
            g => g.s.g.s.g.FooID, 
            sv => sv.FooID, 
            (g, v) => 
                new  
                {
                    g = g, 
                    v = v
                }
        )
        .SelectMany (
            s => s.v.DefaultIfEmpty (),  
            (s, v) => 
                new  
                {
                    s = s, 
                    v = v
                }
        )
        .OrderBy (a => a.s.g.s.g.v.Text)
        .ThenBy (a => a.s.g.v.Text)
        .ThenByDescending (a => a.v.Date)
        .Select (a => a.s.g.s.g.s.g);

Answer 1

If you're having trouble figuring out how to generate the expressions, you could always get an assist from the compiler. What you could do is declare an lambda expression with the types you are going to query with and write the lambda. The compiler will generate the expression for you and you can examine it to see what expressions make up the expression tree.

e.g., your expression is equivalent to this using the query syntax (or you could use the method call syntax if you prefer)

Expression<Func<IQueryable<Foo>, IQueryable<Bar>, IQueryable>> expr =
    (Foo, Bar) =>
        from foo in Foo
        join bar in Bar on foo.Foo_Id equals bar.Foo_Id into bars
        from bar in bars.DefaultIfEmpty()
        select new
        {
            Foo = foo,
            Bar = bar,
        };

---

To answer your question, you can't really generate an expression that creates an anonymous object, the actual type isn't known at compile time. You can cheat kinda by creating a dummy object and use GetType() to get its type which you could then use to create the appropriate new expression, but that's more of a dirty hack and I wouldn't recommend doing this. Doing so, you won't be able to generate strongly typed expressions since you don't know the type of the anonymous type.

e.g.,

var dummyType = new
{
    foo = default(Foo),
    bars = default(IQueryable<Bar>),
}.GetType();

var fooExpr = Expression.Parameter(typeof(Foo), "foo");
var barsExpr = Expression.Parameter(typeof(IQueryable<Bar>), "bars");
var fooProp = dummyType.GetProperty("foo");
var barsProp = dummyType.GetProperty("bars");
var ctor = dummyType.GetConstructor(new Type[]
{
    fooProp.PropertyType,
    barsProp.PropertyType,
});
var newExpr = Expression.New(
    ctor,
    new Expression[] { fooExpr, barsExpr },
    new MemberInfo[] { fooProp, barsProp }
);
// the expression type is unknown, just some lambda
var lambda = Expression.Lambda(newExpr, fooExpr, barsExpr);

Whenever you need to generate an expression that involves an anonymous object, the right thing to do would be to create an known type and use that in place of the anonymous type. It will have limited use yes but it's a much cleaner way to handle such a situation. Then at least you'll be able to get the type at compile time.

// use this type instead of the anonymous one
public class Dummy
{
    public Foo foo { get; set; }
    public IQueryable<Bar> bars { get; set; }
}

var dummyType = typeof(Dummy);

var fooExpr = Expression.Parameter(typeof(Foo), "foo");
var barsExpr = Expression.Parameter(typeof(IQueryable<Bar>), "bars");
var fooProp = dummyType.GetProperty("foo");
var barsProp = dummyType.GetProperty("bars");
var ctor = dummyType.GetConstructor(Type.EmptyTypes);

var newExpr = Expression.MemberInit(
    Expression.New(ctor),
    Expression.Bind(fooProp, fooExpr),
    Expression.Bind(barsProp, barsExpr)
);
// lambda's type is known at compile time now
var lambda = Expression.Lambda<Func<Foo, IQueryable<Bar>, Dummy>>(
    newExpr,
    fooExpr,
    barsExpr);

Or, instead of creating and using a dummy type, you might be able to use tuples in your expressions instead.

static Expression<Func<T1, T2, Tuple<T1, T2>>> GetExpression<T1, T2>()
{
    var type1 = typeof(T1);
    var type2 = typeof(T2);
    var tupleType = typeof(Tuple<T1, T2>);

    var arg1Expr = Expression.Parameter(type1, "arg1");
    var arg2Expr = Expression.Parameter(type2, "arg2");
    var arg1Prop = tupleType.GetProperty("Item1");
    var arg2Prop = tupleType.GetProperty("Item2");
    var ctor = tupleType.GetConstructor(new Type[]
    {
        arg1Prop.PropertyType,
        arg2Prop.PropertyType,
    });

    var newExpr = Expression.New(
        ctor,
        new Expression[] { arg1Expr, arg2Expr },
        new MemberInfo[] { arg1Prop, arg2Prop }
    );
    // lambda's type is known at compile time now
    var lambda = Expression.Lambda<Func<T1, T2, Tuple<T1, T2>>>(
        newExpr,
        arg1Expr,
        arg2Expr);
    return lambda;
}

Then to use it:

var expr = GetExpression<Foo, IQueryable<Bar>>();