Best practice for parameter: IEnumerable vs. IList vs. IReadOnlyCollection

Question

I get when one would return an IEnumerable from a method—when there's value in deferred execution. And returning a List or IList should pretty much be only when the result is going to be modified, otherwise I'd return an IReadOnlyCollection, so the caller knows what he's getting isn't intended for modification (and this lets the method even reuse objects from other callers).

However, on the parameter input side, I'm a little less clear. I could take an IEnumerable, but what if I need to enumerate more than once?

The saying "Be conservative in what you send, be liberal in what you accept" suggests taking an IEnumerable is good, but I'm not really sure.

For example, if there are no elements in the following IEnumerable parameter, a significant amount of work can be saved in this method by checking .Any() first, which requires ToList() before that to avoid enumerating twice.

public IEnumerable<Data> RemoveHandledForDate(IEnumerable<Data> data, DateTime dateTime) {
   var dataList = data.ToList();

   if (!dataList.Any()) {
      return dataList;
   }

   var handledDataIds = new HashSet<int>(
      GetHandledDataForDate(dateTime) // Expensive database operation
         .Select(d => d.DataId)
   );

   return dataList.Where(d => !handledDataIds.Contains(d.DataId));
}

So I'm wondering what is the best signature, here? One possibility is IList<Data> data, but accepting a list suggests that you plan to modify it, which is not correct—this method doesn't touch the original list, so IReadOnlyCollection<Data> seems better.

But IReadOnlyCollection forces callers to do ToList().AsReadOnly() every time which gets a bit ugly, even with a custom extension method .AsReadOnlyCollection. And that's not being liberal in what is accepted.

What is best practice in this situation?

This method is not returning an IReadOnlyCollection because there may be value in the final Where using deferred execution as the whole list is not required to be enumerated. However, the Select is required to be enumerated because the cost of doing .Contains would be horrible without the HashSet.

I don't have a problem with calling ToList, it just occurred to me that if I need a List to avoid multiple enumeration, why do I not just ask for one in the parameter? So the question here is, if I don't want an IEnumerable in my method, should I really accept one in order to be liberal (and ToList it myself), or should I put the burden on the caller to ToList().AsReadOnly()?

Further Information for those unfamiliar with IEnumerables

The real problem here is not the cost of Any() vs. ToList(). I understand that enumerating the entire list costs more than doing Any(). However, assume the case that the caller will consume all items in the return IEnumerable from the above method, and assume that the source IEnumerable<Data> data parameter comes from the result of this method:

public IEnumerable<Data> GetVeryExpensiveDataForDate(DateTime dateTime) {
    // This query is very expensive no matter how many rows are returned.
    // It costs 5 seconds on each `.GetEnumerator` call to get 1 value or 1000
    return MyDataProvider.Where(d => d.DataDate == dateTime);
}

Now if you do this:

var myData = GetVeryExpensiveDataForDate(todayDate);
var unhandledData = RemoveHandledForDate(myData, todayDate);
foreach (var data in unhandledData) {
   messageBus.Dispatch(data); // fully enumerate
)

And if RemovedHandledForDate does Any and does Where, you'll incur the 5 second cost twice, instead of once. This is why you should always take extreme pains to avoid enumerating an IEnumerable more than once. Do not rely on your knowledge that in fact it's harmless, because some future hapless developer may call your method some day with a newly implemented IEnumerable you never thought of, which has different characteristics.

The contract for an IEnumerable says that you can enumerate it. It does NOT promise anything about the performance characteristics of doing so more than once.

In fact, some IEnumerables are volatile and won't return any data upon a subsequent enumeration! Switching to one would be a totally breaking change if combined with multiple enumeration (and a very hard to diagnose one if the multiple enumeration was added later).

Don't do multiple enumeration of an IEnumerable.

If you accept an IEnumerable parameter, you are in effect promising to enumerate it exactly 0 or 1 times.

Answer 1

IReadOnlyCollection<T> adds to IEnumerable<T> a Count property and the corresonding promise that there is no deferred execution. It would be the appropriate parameter to ask for, if the parameter is where you want to tackle this problem.

However, I suggest asking for IEnumerable<T>, and calling ToList() in the implementation itself instead.

Observation: Both approaches have the drawback that the multiple enumeration may at some point be refactored away, rendering the parameter change or ToList() call redundant, which we may overlook. I do not think this can be avoided.

The case does speak for calling ToList() in the method body: Since the multiple enumeration is an implementation detail, avoiding it should be an implementation detail as well. This way, we avoid affecting the API. We also avoid changing back the API if the multiple enumeration ever gets refactored away. We also avoid propagating the requirement through a chain of methods, all of which would have to ask for an IReadOnlyCollection<T> just because of our multiple enumeration.

If you are concerned with the overhead of creating extra lists (when the output already is a list or so), Resharper suggests the following approach:

param = param as IList<SomeType> ?? param.ToList();

Of course, we can do even better, because we only need to protect against deferred execution - no need for a full-blown IList<T>:

param = param as IReadOnlyCollection<SomeType> ?? param.ToList();

Answer 2

There are definitely ways around that will let you accept IEnumerable<T>, only enumerate once and make sure you don't query the database multiple times. Solutions I can think of:

• instead of using Any and Where you could use the enumerator directly. Call MoveNext instead of Any to see if there are any items in the collection, and manually iterate further in after making your database query.
• use Lazy to initialize your HashSet.

The first one seems ugly, the second one might actually make a lot of sense:

public IEnumerable<Data> RemoveHandledForDate(IEnumerable<Data> data, DateTime dateTime)
{
    var ids = new Lazy<HashSet<int>>(
        () => new HashSet<int>(
       GetHandledDataForDate(dateTime) // Expensive database operation
          .Select(d => d.DataId)
    ));

    return data.Where(d => !ids.Value.Contains(d.DataId));
}

Answer 3

You can take an IEnumerable<T> in the method, and use a CachedEnumerable similar to the one here to wrap it.

This class wraps an IEnumerable<T> and makes sure that it is only enumerated once. If you try to enumerate it again, it yield items from the cache.

Please note that such wrapper does not read all items from the wrapped enumerable immediately. It only enumerates individual items from the wrapped enumerable as you enumerate individual items from the wrapper, and it caches the individual items along the way.

This means that if you call Any on the wrapper, only a single item will be enumerated from the wrapped enumerable, and then such item will be cached.

If you then use the enumerable again, it will first yield the first item from the cache, and then continue enumerating the original enumerator from where it left.

You can do something like this to use it:

public IEnumerable<Data> RemoveHandledForDate(IEnumerable<Data> data, DateTime dateTime)
{
    var dataWrapper = new CachedEnumerable(data);
    ...
}

Notice here that the method itself is wrapping the parameter data. This way, you don't force consumers of your method to do anything.

Answer 4

I don't think this can be solved just by changing the input types. If you want to allows more general structures than List<T> or IList<T> then you have to decide if/how to handle these possible edge cases.

Either plan for the worst case and spend a little time/memory creating a concrete data structure, or plan for the best case and risk the occasional query getting executed twice.

You might consider documenting that the method enumerates the collection multiple times so that the caller can decide if they want to pass in an "expensive" query, or hydrate the query before calling the method.

Answer 5

I would argue that IEnumerable<T> is a good option for an argument type. It is a simple, generic and easy to provide structure. There is nothing inherent about the IEnumerable contract that implies that one should only ever iterate it once.

In general, the performance cost for testing .Any() probably isn't high but, of course, cannot be guaranteed to be so. In the circumstances you describe, it could obviously be the case that iterating the first element has considerable overhead but this is by no means universal.

Changing the parameter type to something like IReadOnlyCollection<T> or IReadOnlyList<T> is an option but probably only a good one in the circumstance that some or all of the properties/methods provided by that interface are required.

If you don't need that functionality and instead want to guarantee that your method only iterates the IEnumerable once, you can do so by calling .ToList() or by turning it into some other appropriate type of collection but that is an implementation detail of the method itself. If the contract that you are designing requires "something which can be iterated" then IEnumerable<T> is a very appropriate choice.

Your method has the power to make guarantees about how many times any collection will be iterated, you don't need to expose that detail beyond the boundaries of your method.

By contrast, if you do choose to repeatedly enumerate an IEnumerable<T> inside your method then you must also take into consideration every eventuality which could be a result of that choice, for instance potentially getting different results in different circumstances due to deferred execution.

That said, as a point of best practise, I think it makes sense to try to avoid any side-effects in IEnumerables returned by your own code to the maximum extent possible - languages like Haskell can make use of lazy evaluation throughout safely because they go to great pains to avoid side effects. If nothing else, people who consume your code might not be as dilligent as you in guarding against multiple enumeration.