7

I want to get a list of distinct values from my OData endpoint. But distinct or group by isn't supported yet.

My URI query looks something like this

GET /odata/Products?$select=foo & $top=10 & $count=true & distinct=true

My Controller

[EnableQuery]
public IQueryable<FooBarBaz> Get(ODataQueryOptions<FooBarBaz> queryOptions, bool distinct)
{
        //I've tried the following
        return Repository.AsQueryable().Distinct();

        // and
        return Repository.AsQueryable().GroupBy(x => x.Foo);

        // and
        IQueryable query = queryOptions.ApplyTo(Repository.AsQueryable());
        return query.Distinct(); // Can't call .Distinct() here
}

None work :(

Snæbjørn
  • 10,322
  • 14
  • 65
  • 124
  • The component [AdaptiveLINQ](http://www.adaptivelinq.com) can help you. Disclaimer: I'm the AdaptiveLINQ developer. – nlips May 11 '15 at 16:35
  • The major reason this isn't supported is that the _Entity_ records served by an OData controller by definition _MUST_ be unique. You declare in the configuration the column that provides the unique key and if there is a unique column, then the records will always be distinct. – Chris Schaller Dec 07 '21 at 04:50

2 Answers2

12

Because you have specified the EnableQuery attribute, you can use $apply to groupby your distinct fields, without having to add any custom functions or parameters, you get this for free out of the box:

GET /odata/Products?$apply=groupby((foo))&top=10&$count=true

This is simple OData v4 standard syntax that doesn't require any code modification to implement. Don't go changing each controller that you want to support a distinct query on, you can't know 100% in advance which controllers your client apps might want this functionality on, so use the functionality that is provided before you start customisations.

Of course there is a caveat to this approach that do not make it viable 100% of the time:

  • $filter and $orderby can only operate on the fields specified in your group by clause

This may require you to include additional fields into your grouping statement and for some complex filtering the resultant dataset may not be satisfactory, in cases such as these we found it easier to support passing in an additional pre-filter parameter through HTTP header that can be applied to the query before the passed in query options are applied, note that this was only necessary because our filter conditions were tenancy and security related and so the resultant data set had many more duplicate entries if you ignored the security descriptors.

Just for fun, here is our custom GET function that applies the pre-filter if it is passed in:

[EnableQuery]
public IQueryable<FooBarBaz> Get(ODataQueryOptions<FooBarBaz> queryOptions, bool distinct)
{
    DbQuery<FooBarBaz> query = Repository;
    query = this.ApplyUserPolicy(query);
    return Ok(query);
}

The following is implemented in a base class so that we don't have it in each controller:

/// <summary>
/// Apply default user policy to the DBQuery that will be used by actions on this controller.
/// The big one we support here is X-Filter HTTP headers, so now you can provide top level filtering in the header of the request 
/// before the normal OData filter and query parameters are applied.
/// This is useful when you want to use $apply and $filter together but on separate sets of conditions.
/// </summary>
/// <param name="dataTable">DBQuery to apply the policy to</param>
/// <returns>Returns IQueryable entity query ready for processing with the headers applied (if any)</returns>
private IQueryable<TEntity> ApplyUserPolicy(DbQuery<TEntity> dataTable)
{
    // Proprietary Implementation of Security Tokens
    //var tokenData = SystemController.CurrentToken(Request);
    //IQueryable<TEntity> query = ApplyUserPolicy(dataTable, tokenData);
    IQueryable<TEntity> query = dataTable.AsQueryable();

    // Now try and apply an OData filter passed in as a header.
    // This means we are applying a global filter BEFORE the normal OData query params
    // ... we can filter before $apply and group by

    System.Collections.Generic.IEnumerable<string> filters = null;
    if (Request.Headers.TryGetValues("X-Filter", out filters))
    {
        foreach (var filter in filters)
        {
            //var expressions = filter.Split(',');
            //foreach (var expression in expressions)
            {
                var expression = filter;
                Dictionary<string, string> options = new Dictionary<string, string>()
                {
                    { "$filter"  , expression },
                };

                var model = this.Request.ODataProperties().Model;
                IEdmNavigationSource source = model.FindDeclaredEntitySet(this.GetEntitySetName());
                var type = source.EntityType();
                Microsoft.OData.Core.UriParser.ODataQueryOptionParser parser
                    = new Microsoft.OData.Core.UriParser.ODataQueryOptionParser(model, type, source, options);
                var filterClause = parser.ParseFilter();     // parse $filter 

                FilterQueryOption option = new FilterQueryOption(expression, new ODataQueryContext(model, typeof(TEntity), this.Request.ODataProperties().Path), parser);
                query = (IQueryable<TEntity>)option.ApplyTo(query, new ODataQuerySettings());
            }
        }
    }


    return query;
}

If nothing else, it's cheaper that trying to sell AdaptiveLINQ to your manager :)

Chris Schaller
  • 13,704
  • 3
  • 43
  • 81
  • I was able to get a Distinct response just by the $apply and groupby in my filter. Its likely you aren't getting more thanks because of the other verbose code in your response. I wanted to highlight to others that the $apply and groupby returns a distinct list. – lyndon hughey Nov 07 '17 at 22:33
  • 1
    true but $apply only works if you do not also need to $filter, unless of course your $filter is operating on the results columns in your $apply, and even then there are restrictions. So while I had a similar problem to OP, when I tried to implement my simple $apply in my own code first I quickly decided that the answer needed to be more robust. It was fun, hopefully someone from the ODataLib team takes notice and adds some similar support for us later – Chris Schaller Nov 07 '17 at 22:40
  • @ChrisSchaller - I have been fooling around with odata, as I want to perform a distinct query with it, but I also want pagination. Meaning, let's say that I have 330 distinct values, and with a page size of 100, I want 4 pages. And to do this, I'm executing my queryable (that's getting the odata applied to) using .Skip.Take.ToList, but here is what I'm finding: Executing the first page of 100 it's only getting the distinct values for that page and not the superset before paging. Is there a way to groupby on the superset and then page? – JustLooking Dec 07 '21 at 02:47
  • 2
    I would create a custom `IQueryable` _Function_ endpoint to serve the distinct recordset in that case, or not page the data. 330 values is not a lot, you could bring that back in a single hit and manage the paging on the client if you really wanted a paged view. The need for `Distinct` usually indicates you do not have a sufficiently normalized data structure or you are executing the query from the wrong controller, serve the query from a controller that already provides the distinct records. – Chris Schaller Dec 07 '21 at 02:53
  • @ChrisSchaller - Actually Distinct has a purpose. We have a lot of screens with grids. And each grid will have columns tied to the database. One of the features we have is "filtering". So, you can select any column and then "browse" the data associated with that column for data values to be part of your filter. Hence, the need for distinct. As for the 330, that was an example, all I know is that this has been very difficult! Fighting through some things now. – JustLooking Dec 08 '21 at 03:09
8

The best solution to solve the problem by defining an collection Action on the resource.

First Step : configure the 'Distinct' action in WebApiConfig.cs

ODataConventionModelBuilder builder = new ODataConventionModelBuilder();
builder.EntitySet<FooBarBaz>("FooBarBazs");//Resource Name

ActionConfiguration Distinct = builder.Entity<FooBarBaz>().Collection.Action("Distinct");//Name of the action method
Distinct.ReturnsCollectionFromEntitySet<FooBarBaz>("FooBarBazs");//Return type of action
Distinct.Parameter<string>("On");//Property on which collection is filtered as Distinct

config.Routes.MapODataRoute("odata", "odata", builder.GetEdmModel());

Second Step : Add the Action in FooBarBazsController.cs which returns the collection of distinct entities

[EnableQuery]//enable the $select,$expend Queries
[HttpPost]//All the action methods are of post type in Web api
public IQueryable<FooBarBaz> Distinct(ODataActionParameters parameters)
{
        string on = "";
        if (!ModelState.IsValid)
        {
            throw new HttpResponseException(HttpStatusCode.BadRequest);
        }

        try
        {
             on = parameters["On"] as string;
        }
        catch (NullReferenceException ex)
        {
            HttpResponseMessage message = new HttpResponseMessage(HttpStatusCode.BadRequest);
            message.Content = new StringContent("{\"Error\":\"Invalid Query -> On property is not defined\"}");
            throw new HttpResponseException(message);
        }
        catch (Exception ex)
        {
            throw new HttpResponseException(HttpStatusCode.BadRequest);
        }


        PropertyInfo[] props = new FooBarBaz().GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance);
        var isPropertyExist = false;
        for (int i = 0; i < props.Length; i++)
        {
            if (props[i].Name.Equals(on))
            {
                isPropertyExist = true;
                break;
            }
        }


        if (isPropertyExist)
        {
            var fooBarBazCollection = db.fooBarBazs.GroupBy(GetGroupKey(on)).Select(g => g.FirstOrDefault());//Select the Distinct Entity on the basis of a property
            return fooBarBazCollection ;
        }
        else
        {
            HttpResponseMessage message = new HttpResponseMessage(HttpStatusCode.BadRequest);
            message.Content = new StringContent("{\"Error\":\"Property '"+on+"' Not Exist}");
            throw new HttpResponseException(message);
        }
}

Third Step : Add a static method which returns an Expression for groupby on the basis of Property Name.

private static Expression<Func<fooBarBaz, string>> GetGroupKey(string property)
    {
        var parameter = Expression.Parameter(typeof(fooBarBaz));
        var body = Expression.Property(parameter, property);
        return Expression.Lambda<Func<fooBarBaz, string>>(body, parameter);
    } 

Now Build the project and You can query the Resource like this

POST /odata/FooBarBazs/Distinct HTTP/1.1
Host: localhost:9360
Content-Type: application/json
Cache-Control: no-cache
Postman-Token: 6d174086-7b97-76a2-679c-4dab3dfb5938

{"On":"PropertyName"} 

And can also use the $select and $expend like this

POST /odata/FooBarBazs/Distinct?$select=PropertyName1,PropertyName2 HTTP/1.1
Host: localhost:9360
Content-Type: application/json
Cache-Control: no-cache
Postman-Token: 6d174086-7b97-76a2-679c-4dab3dfb5938

{"On":"PropertyName"} 

I hope this solve the problem. +1 if it do.

Ravi Kumar Mistry
  • 1,063
  • 1
  • 13
  • 24