"Nullable object must have a value" exception after checking for null on a non-primitive/non-struct object

Question

I'm getting Nullable object must have a value after checking for null on a regular object, after a null check. I've found various questions, mostly regarding linq-to-sql, having the same problem but always with nullable primitive types (as in bool? or DateTime?).

The line causing the exception in my case looks like this:

myDataContext.Orders.Where(y => customer.Address == null || (string.IsNullOrEmpty(customer.Address.Street) || y.Customers.Addresses.Street == customer.Address.Street)))

customer class looks like this:

public class Customer
{
    private Address address = null;
    public Address Address{get{return address;} set{address=value;}}
}

address property looks like this:

public class Address
{
    private string street = null;
    public string Street{get{return street ;} set{street =value;}}
}

If I replace above code line with this:

string custStreet = null;
if (customer.Address != null)
{
    custStreet = customer.Address.Street;
}

myDataContext.Orders.Where(y =>(customer.Address == null || (string.IsNullOrEmpty(custStreet) || y.Customers.Addresses.Street == custStreet)))

it runs fine. I don't undestand the reason for that. I also don't want to define countless variables before executing the Lambda statement itself.

Please also note that above Lambda statement is part of a much bigger Lambda Where clause that contains a few more of such statements. I know I could work with Expression Trees but having coded this far, I really don't want to switch now.

edit

as the question was answered, I'm going to tell you how I worked around it: I build myself a recursive property initializer. Everything that is not a string, a list/array or a primitive type is thrown against the Activator class. I got the idea from here and made a few changes to it (basically, ignore everything that doesn't need to be initialized and instead of Activator.CreateInstance(Type.GetType(property.PropertyType.Name)); I used Activator.CreateInstance(property.PropertyType)); I'm not even sure if the version used in the original question would work or why anyone would want to use it.)

Check the exception stack trace - it might be a bug in the query provider you are using. — Ivan Stoev, Apr 27 '16 at 14:12
That line is incomplete though. What is it? An `if`? A boolean assigned to a var? On that note... I don't see any lambda expressions in there. That's just some boolean operations. — Nyerguds, Apr 27 '16 at 14:13
@IvanStoev `at Devart.Data.Linq.Engine.bp.c.a(Expression A_0)` it's completely useless. It could be a problem with the provider, though. — Steffen Winkler, Apr 27 '16 at 14:16
@Nyerguds not sure what you mean with the first part 'What is it?'. In any case, I added some code to the line so it makes more sense, I guess. Even though it shouldn't really matter. — Steffen Winkler, Apr 27 '16 at 14:18
That's what I had in mind. Most likely the provider is trying to reduce constant predicates and does that incorrectly. — Ivan Stoev, Apr 27 '16 at 14:21
Hmm. So the intent is that if either `customer.Address` or `customer.Address.Street` are null, it would just return everything from `Orders`? An optional filter? Could be it has a problem with never getting to the part where y is actually used, then. But, certainly an odd case. — Nyerguds, Apr 27 '16 at 14:22
I don;t think you've shown what is _actually_ causing the exception. You do not have any nullable value types in that expression. Can you post the _entire_ method instead of just snippets? — D Stanley, Apr 27 '16 at 14:23
@Nyerguds correct. It's basically a large statement for a UI search function. And it definetly hits the `y`, you can see that in the later part of my question where I provide a workaround for the solution that works. — Steffen Winkler, Apr 27 '16 at 14:24
@DStanley that's the entire point: I've no primitive, nullable types in that entire statement, anywhere. Well, in theory I've them but I removed them from the statement for now. Exception remains. And it's caused by that line, look at the second part of my question. I explain how I can work around that bug in theory and that it works if I do that. — Steffen Winkler, Apr 27 '16 at 14:25
@SteffenWinkler You don't have nullable value type, but you do have nullable types, and you're evaluating them to a value when they're null, hence the error. — Servy, Apr 27 '16 at 14:29
@Servy I disagree. The query provider is not forced to translate the exact expression to SQL. It can see that `customer.Address == null` is a constantly true and eliminate the whole `Where`. Similar to what you'll do manually using `if (condition) query = query.Where(...)`. The problem is though that they **don't** do such optimizations. — Ivan Stoev, Apr 27 '16 at 15:10
@IvanStoev Saying that it erroring is incorrect is to say that it's *obligated* to make such an optimization. It is not. You'll find that pretty much all query providers, as a rule, won't do much at all in the way of optimizations, and they certainly aren't *expected* to perform them. — Servy, Apr 27 '16 at 15:14
@Servy Agreed. Probably I didn't state it correctly - it would be nice if they do that because I'm seeing a lot of similar questions on SO (not NRE related) - for instance [LINQ to Entities conditionals give weird results](http://stackoverflow.com/questions/36761694/linq-to-entities-conditionals-give-weird-results) and in general for building dynamic `Where` clauses. While the general answer is - don't do that, use `if`s or predicate builders, it would be nice if there is a (sort of) general solution. I'm thinking of doing something in that regard myself. — Ivan Stoev, Apr 27 '16 at 15:25

score 2 · Answer 1 · edited May 23 '17 at 12:23

Contrary to what I wrote in the comments, the problem is that query providers does not even try to reduce the predicate expressions by eliminating the constant parts. As @Servy correctly stated in the comments, they are not forced to do that, and speaking generally there might be a technical reason not doing it, but in reality people tend to use such conditions in their query expressions and expect them to work as if they are evaluated in LINQ to Objects.

I've seen many questions with similar usage, the last being LINQ to Entities conditionals give weird results, and the "standard" comment/answer is - use chained Where with ifs or some predicate builder. Then I start thinking - ok, the providers don't do that, so why don't we do that ourselves then - after all, we are developers and can write (some) code. So I've ended up with the following extension method which uses ExpressionVisitor to modify the query expression tree. I was thinking to post it to the linked question, but since I've get somehow involved in this thread, here you go:

public static class QueryableExtensions
{
    public static IQueryable<T> ReduceConstPredicates<T>(this IQueryable<T> source)
    {
        var reducer = new ConstPredicateReducer();
        var expression = reducer.Visit(source.Expression);
        if (expression == source.Expression) return source;
        return source.Provider.CreateQuery<T>(expression);
    }

    class ConstPredicateReducer : ExpressionVisitor
    {
        private int evaluateConst;
        private bool EvaluateConst { get { return evaluateConst > 0; } }
        private ConstantExpression TryEvaluateConst(Expression node)
        {
            evaluateConst++;
            try { return Visit(node) as ConstantExpression; }
            catch { return null; }
            finally { evaluateConst--; }
        }
        protected override Expression VisitUnary(UnaryExpression node)
        {
            if (EvaluateConst || node.Type == typeof(bool))
            {
                var operandConst = TryEvaluateConst(node.Operand);
                if (operandConst != null)
                {
                    var result = Expression.Lambda(node.Update(operandConst)).Compile().DynamicInvoke();
                    return Expression.Constant(result, node.Type);
                }
            }
            return EvaluateConst ? node : base.VisitUnary(node);
        }
        protected override Expression VisitBinary(BinaryExpression node)
        {
            if (EvaluateConst || node.Type == typeof(bool))
            {
                var leftConst = TryEvaluateConst(node.Left);
                if (leftConst != null)
                {
                    if (node.NodeType == ExpressionType.AndAlso)
                        return (bool)leftConst.Value ? Visit(node.Right) : Expression.Constant(false);
                    if (node.NodeType == ExpressionType.OrElse)
                        return !(bool)leftConst.Value ? Visit(node.Right) : Expression.Constant(true);
                    var rightConst = TryEvaluateConst(node.Right);
                    if (rightConst != null)
                    {
                        var result = Expression.Lambda(node.Update(leftConst, node.Conversion, rightConst)).Compile().DynamicInvoke();
                        return Expression.Constant(result, node.Type);
                    }
                }
            }
            return EvaluateConst ? node : base.VisitBinary(node);
        }
        protected override Expression VisitConditional(ConditionalExpression node)
        {
            if (EvaluateConst || node.Type == typeof(bool))
            {
                var testConst = TryEvaluateConst(node.Test);
                if (testConst != null)
                    return Visit((bool)testConst.Value ? node.IfTrue : node.IfFalse);
            }
            return EvaluateConst ? node : base.VisitConditional(node);
        }
        protected override Expression VisitMember(MemberExpression node)
        {
            if (EvaluateConst || node.Type == typeof(bool))
            {
                var expressionConst = node.Expression != null ? TryEvaluateConst(node.Expression) : null;
                if (expressionConst != null || node.Expression == null)
                {
                    var result = Expression.Lambda(node.Update(expressionConst)).Compile().DynamicInvoke();
                    return Expression.Constant(result, node.Type);
                }
            }
            return EvaluateConst ? node : base.VisitMember(node);
        }
        protected override Expression VisitMethodCall(MethodCallExpression node)
        {
            if (EvaluateConst || node.Type == typeof(bool))
            {
                var objectConst = node.Object != null ? TryEvaluateConst(node.Object) : null;
                if (objectConst != null || node.Object == null)
                {
                    var argumentsConst = new ConstantExpression[node.Arguments.Count];
                    int count = 0;
                    while (count < argumentsConst.Length && (argumentsConst[count] = TryEvaluateConst(node.Arguments[count])) != null)
                        count++;
                    if (count == argumentsConst.Length)
                    {
                        var result = Expression.Lambda(node.Update(objectConst, argumentsConst)).Compile().DynamicInvoke();
                        return Expression.Constant(result, node.Type);
                    }
                }
            }
            return EvaluateConst ? node : base.VisitMethodCall(node);
        }
    }
}

With that extension method in place, all you need is to insert .ReduceConstPredicates() at the end of your queries (before AsEnumerable(), ToList and similar):

var query = myDataContext.Orders
    .Where(y => customer.Address == null || string.IsNullOrEmpty(customer.Address.Street) || y.Customers.Addresses.Street == customer.Address.Street)
    .ReduceConstPredicates();

score 1 · Accepted Answer · answered Apr 27 '16 at 14:28

1

The value of the expression customer.Address.Street must be evaluated to its value *before the query can be translated into SQL. That expression cannot be left in the underlying SQL for the database to possibly, or possibly not, evaluate to a value. The query provider has to evaluate it unconditionally in order to determine what the SQL should look like. So yes, you do need to perform the null check outside of the expression. There are of course any number of ways you could do so, but that null checking logic does need to be outside of the expression the query provider translates.

answered Apr 27 '16 at 14:28

Servy

202,030
26
332
449

that'd be exactly what I'm fearing. But shouldn't the provider see that my local `Address` object is null and just ignore everything after the `||` when translating to SQL? – Steffen Winkler Apr 27 '16 at 14:31
@SteffenWinkler The purpose of the provider is to translate the entire expression into SQL, not to try to evaluate the expression on items that it doesn't even have. This isn't linq to objects, it's not executing the code as code, it needs to translate *the whole thing* into SQL. – Servy Apr 27 '16 at 14:32
but that's my point exactly: It's not running into a `null` on the database. My local `Address` object that the provider has to know to build the SQL statement is null. Not anything on the database. The `customer` object is defined and declared outside of the Lambda statement. – Steffen Winkler Apr 27 '16 at 14:33
Ah, right. This isn't just linq; it's an actual database query, isn't it? It's not running into null on the database, it's running into null in the data required to build the query. – Nyerguds Apr 27 '16 at 14:34
@Nyerguds the result would be a SQL statement, yes. – Steffen Winkler Apr 27 '16 at 14:35
@SteffenWinkler Yes, that's correct. The query provider needs to evaluate `customer.Address.Street` in order to use that value in the generated SQL statement that is sent to the database, because that value cannot be evaluated by the database. A null value is de-referenced when attempting to evaluate that expression and generate the SQL, hence the error. – Servy Apr 27 '16 at 14:40
@Servy ah, thank you for that explanation. #Answered – Steffen Winkler Apr 27 '16 at 14:42

"Nullable object must have a value" exception after checking for null on a non-primitive/non-struct object

2 Answers2

Linked