2

I have a problem trying to get the count out of the following query:

var usersView = PopulateUsersView(); //usersView is an IQueryable object
var foo = usersView.Where(fields => fields.ConferenceRole.ToLower().Contains("role"));

Where UsersView is a class which is populated from an EF entity called users (refer to the first line in the code above)

This is the class definition for the UsersView class:

public class UsersView
{
    public int UserId { get; set; }
    public string Title { get; set; }
    public string Name { get; set; }
    public string Surname { get; set; }
    public string Street1 { get; set; }
    public string Street2 { get; set; }
    public string City { get; set; }
    public string PostCode { get; set; }
    public string CountryName { get; set; }
    public string WorkPlaceName { get; set; }
    public string Gender { get; set; }
    public string EMail { get; set; }
    public string Company { get; set; }
    public string RoleName { get; set; }
    public string ConferenceRole { get; set; }
}

As I said trying to execute the line foo.Count() returns Null Exception and this might be because the ConferenceRole column allows Null in the database.

Now what I can't understand is that when I invoke the same query directly on the ObjectQuery the Count of records (i.e. invoking foo2.Count()) is returned without any exceptions.

var foo2 = entities.users.Where(fields => fields.ConferenceRole.ToLower().Contains("role"));

Is it possible to the same query above but using the IQueryable usersView object instead?

(It is crucial for me to use the usersView object rather than directly querying the entities.users entity)

EDIT

Below is the code from the PopulateUsersView method

private IQueryable<UsersView> PopulateUsersView()
    {
        using (EBCPRegEntities entities = new EBCPRegEntities())
        {
            var users = entities.users.ToList();
            List<UsersView> userViews = new List<UsersView>();
            foreach (user u in users)
            {
                userViews.Add(new UsersView()
                {
                    UserId = u.UserId,
                    Title = u.Title,
                    Name = u.Name,
                    Surname = u.Surname,
                    Street1 = u.Street1,
                    Street2 = u.Street2,
                    City = u.City,
                    PostCode = u.Post_Code,
                    CountryName = u.country.Name,
                    WorkPlaceName = u.workplace.Name,
                    Gender = u.Gender,
                    EMail = u.E_Mail,
                    Company = u.Company,
                    RoleName = u.roles.FirstOrDefault().Name,
                    ConferenceRole = u.ConferenceRole
                });
            }
            return userViews.AsQueryable();
        }
    }

Thanks

UPDATE...

Thanks guys I finally found a good answer to the difference between the IQueryable and the ObjectQuery objects.

As a solution I am checking if the ConferenceRole is null and then checking with the contains method as many of you guys have said.

Ryan
  • 265
  • 1
  • 6
  • 17
  • I suggest you do a comparison of the two approaches: What is the content of `ConferenceRole` for entities where it is NULL in the database? – Daniel Hilgarth Jul 14 '11 at 08:04
  • It's string and by default it's created as null in the database. – Ryan Jul 14 '11 at 08:07
  • That was not what I was asking. Obviously, ConferenceRole is null when you access it via the usersView, but not when you access it directly from the database. So what is it then? – Daniel Hilgarth Jul 14 '11 at 08:08
  • You mean using the Except extension method? between all records returned by the IQueryable (usersView) object and the filtered result set of the ObjectQuery? – Ryan Jul 14 '11 at 08:09
  • Modify your query to add a `null` check `usersView.Where(fields => fields.ConferenceRole != null && ....);` – Clayton Jul 14 '11 at 08:09
  • no... Go to the database and look for a row where `ConferenceRole` is empty. Take the ID of the row and compare `entities.users.Where(x=> x.ID = foundID).First().ConferenceRole` to `usersView.Where(x=> x.ID = foundID).First().ConferenceRole` – Daniel Hilgarth Jul 14 '11 at 08:10
  • @Daniel Hilgarth The value will most likely be the same, LinqToSql make the difference here, check my answer. – ntziolis Jul 14 '11 at 08:47
  • @ntziolis: This sounds plausible +1 – Daniel Hilgarth Jul 14 '11 at 08:49

2 Answers2

2

My guess is that your PopulateUsersView() method is actually executing a query and returning an IQueryable Linq-to-Objects object - while the foo2 line executes the query only in the SQL layer. If this is the case, the obviously PopulateUsersView() is going to be quite an inefficient way to perform the Count

To debug this:


Update

@Ryan - thanks for posting the code to PopulateUsersView

Looks like my guess was right - you are doing a query which gets the whole table back into a List - and its this list that you then query further using Linq2Objects.

@ntziolis has provided one solution to your problem - by testing for null before doing the ToLower(). However, if your only requirement is to Count the non-empty items list, then I recommend you look at changing the PopulateUsersView method or changing your overall design. If all you need is a Count then it would be much more efficient to ensure that the database does this work and not the C# code. This is espeically the case if the table has lots of rows - e.g. you definitely don't want to be pulling 1000s of rows back into memory from the database.


Update 2

Please do consider optimising this and not just doing a simple != null fix.

Looking at your code, there are several lines which will cause multiple sql calls:

  • CountryName = u.country.Name
  • WorkPlaceName = u.workplace.Name
  • RoleName = u.roles.FirstOrDefault().Name

Since these are called in a foreach loop, then to calculate a count of ~500 users, then you will probably make somewhere around 1501 SQL calls (although some roles and countries will hopefully be cached), returning perhaps a megabyte of data in total? All this just to calculate a single integer Count?

Stuart
  • 66,722
  • 7
  • 114
  • 165
  • @Ryan - thanks - I've added an update to my answer. If Users is going to be a large table then do not just apply a simple `!= null` fix here - this will hurt you in performance terms. – Stuart Jul 14 '11 at 09:17
  • Users will be maximum of 500 so it's not that much. Thanks for the tip though – Ryan Jul 14 '11 at 09:28
  • @Ryan - unless you are using that list for other reasons, then even at 500 users you should still push this `Count` down to the database - it will optimize this Count query and will be able to return the answer much more quickly than the large fetch and the c# code will work. If you can, then try to avoid pulling back maybe 500kB of data and executing 500 loops of c# code when the database could just return you a single integer. – Stuart Jul 14 '11 at 09:38
  • Have added an "update 2" to explain my performance concerns - promise I'll stop now :) – Stuart Jul 14 '11 at 09:44
  • I like your thinking Stuart....but this event is not there just for a Count call but this will get passed to a small framework for jqGrid to populate the grid with the data using on the fly expressions. – Ryan Jul 14 '11 at 12:20
  • The reason why I did this was because users entity have some columns which are polled using NavigationProperties inside the EF and the jqGrid does not know how to handle these relationships to get referenced values rather than the id's for example I will be interested in showing users.country.name rather than users.countryId so to do this I had to create a "View" in this case UsersView to flatten all the joined entities within a single object – Ryan Jul 14 '11 at 12:23
  • @Ryan - a quick way to improve the performance would be by putting the conversion inline - before the `ToList()` - you can do this using `.Select(x => new UsersView() { ... }).ToList()` - this will improve the conversion into SQL and allow you to keep your view with names rather than id's. If in doubt check using the EF-SQL trace to see what is being actually executed. – Stuart Jul 14 '11 at 12:47
  • Thanks for all these performance tips will try this out. But first I will look into EF-SQL trace (never used them before although I knew something like that would exist) – Ryan Jul 14 '11 at 12:51
  • @Stuart I'm pretty sure in-lining this way will fail, since linq2sql does not know how to convert `new UsersView` into proper sql. What should be possible however is creating anonymous objects that have all the necessary properties on the fly, then `ToList()`and afterwards copy the values from the anonymous into proper `UserView` objects – ntziolis Jul 15 '11 at 16:50
  • @ntziolis - point taken - I think your point applies for EF as well as Linq2Sql. Since this question is EF (I think) then maybe using `Includes()` would also help. The main thing here is definitely to measure and not to assume anyone knows what's happening :) – Stuart Jul 15 '11 at 18:57
2

Try to check whether ConferenceRole is null before calling a method on it:

var foo = usersView.Where(fields => fields.ConferenceRole != null 
    && fields.ConferenceRole.ToLower().Contains("role"));

This will enable you to call the count method on the user view.

So why does it work against the ObjectQuery?

When executing the query against the ObjectQuery, LinqToSql is converting your query into proper sql which does not have problems with null values, something like this (it's sample markup sql only the actual query looks much different, also '=' is used rather than checking for contains):

SELECT COUNT(*) from USERS U WHERE TOLOWER(U.CONFERENCEROLE) = 'role'

The difference to the :NET code is: It will not call a method on an object but merely call a method and pass in the value, therefore no NullReference can occur in this case.

In order to confirm this you can try to force the .NET runtime to execute the SQL prior to calling the where method, by simply adding a ToList() before the .Where()

var foo2 = entities.users.ToList()
    .Where(fields => fields.ConferenceRole.ToLower().Contains("role"));

This should result in the exact same error you have seen with the UserView.

And yes this will return the entire user table first, so don't use it in live code ;)

UPDATE
I had to update the answer since I c&p the wrong query in the beginning, the above points still stand though.

ntziolis
  • 10,091
  • 1
  • 34
  • 50