4

I have a very large data set (1,000,000+ customers) in a data set on a remote computer. This data is stored in a flat file, and looked up by customer number, which is the offset into the file. I want to provide oData access to this collection, but I only want to offer the ability to read a single entity.

Permissions-wise it's easy to get WCF to restrict it to single reads with the following:

config.SetEntitySetAccessRule("Customer", EntitySetRights.ReadSingle);

The problem I have is that I am forced to return an IQueryable interface which seems to require that I gather the whole dataset. Is there something I'm missing, or is there a way to only gather the records required when evaluating the IQueryable, which I know (due to oData permissions) will require a single record fetch?

Mark Stafford - MSFT
  • 4,306
  • 3
  • 17
  • 23
Dale Bruce Hopkins
  • 63
  • 5
  • This is an aside, but how wide are your rows? I assume this dataset would be trivial to load into memory (500MB?), which is something you should consider in your scenario. File-based I/O is a huge performance hit; most databases mitigate this by keeping hot records in memory. – Mark Stafford - MSFT Jun 29 '12 at 03:12

1 Answers1

1

IQueryable and IEnumerable are very different - without going into too much detail, IQueryable allows you to build a query expression that gets executed by the query provider (on the server) and returns only the result (in your case, the single entity). You will still incur the impact of examining the whole dataset on the server side, but I'm not sure how you would possibly avoid that in your scenario.

See the answer here for more details on the difference between IEnumerable and IQueryable: https://stackoverflow.com/a/252789/981304. Specifically, "...IQueryable allows for out-of-memory things like a remote data source, such as a database or web service."

Community
  • 1
  • 1
Mark Stafford - MSFT
  • 4,306
  • 3
  • 17
  • 23
  • Awesome! The I created a class that implements both IQueryable and IEnumerable. When you run LINQ it uses the IQueryable to gather the WHERE portion of the query which in this case is the customer number. The LINQ doesn't actually gather the data, it just gathers the parameters of the query. Then when you create an enumerator is when I actually gather the data. At this point I have access to the conditions so I can return only the results from the server! Thanks again – Dale Bruce Hopkins Sep 14 '12 at 20:51