Divide a large IEnumerable into smaller IEnumerable of a fix amount of item

Question

In order to support an API that only accepts a specific amount of items (5 items), I want to transform a LINQ result into smaller groups of items that always contain that set amount of items.

Supposing the list {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18}

I want to get three smaller lists of a maximum of 5 items each

{1, 2, 3, 4, 5}

{6, 7, 8, 9, 10}

{11, 12, 13, 14, 15}

{16, 17, 18}

How can I do that with LINQ? I'm assuming that it either involves Group or Aggregate, but I'm having trouble figuring how to write that.

http://stackoverflow.com/questions/1349491/how-can-i-split-an-ienumerablestring-into-groups-of-ienumerablestring — diceguyd30, Mar 02 '11 at 19:34
possible duplicate of [Split List into Sublists with LINQ](http://stackoverflow.com/questions/419019/split-list-into-sublists-with-linq) — nawfal, Feb 18 '13 at 10:28

score 17 · Accepted Answer · answered Mar 02 '11 at 19:33

17

Try something like this:

var result = items.Select((value, index) => new { Index = index, Value = value})
                  .GroupBy(x => x.Index / 5)
                  .Select(g => g.Select(x => x.Value).ToList())
                  .ToList();

It works by partitioning the items into groups based on their index in the original list.

answered Mar 02 '11 at 19:33

Mark Byers

811,555
193
1,581
1,452

Reading your code sparked an idea. `int index = 0; var groups = items.GroupBy(x => index++ / 5);` And this work. Thank you. It is always simple as usual. – Pierre-Alain Vigeant Mar 02 '11 at 19:47
3

@Pierre-Alain: Using side-effects in a LINQ query is generally frowned upon; it goes against the design principles of LINQ, which is primarily functional. It'll work to some extent... but you'll get some interesting effects if you evaluate `groups` twice, for example. – Jon Skeet Mar 02 '11 at 19:50
@Pierre The function passed into `GroupBy` having a side-effect is ugly. Linq is about functional and thus side-effect free programming. – CodesInChaos Mar 02 '11 at 19:51
I guess its because I don't understand how the index in the answer is incremented. But I implemented the version given by Mark and it does work, although I'm clueless on how index is assigned. – Pierre-Alain Vigeant Mar 02 '11 at 20:09
2

Guess I just learned about the `Select` overload that incorporate the item index. I'm no longer ignorant on that topic. – Pierre-Alain Vigeant Mar 02 '11 at 20:12

Dan Tao · Answer 2 · 2011-03-02T19:52:35.410

I'd just do something like this:

public static IEnumerable<IEnumerable<T>> TakeChunks<T>(this IEnumerable<T> source, int size)
{
    // Typically you'd put argument validation in the method call and then
    // implement it using a private method... I'll leave that to your
    // imagination.

    var list = new List<T>(size);

    foreach (T item in source)
    {
        list.Add(item);
        if (list.Count == size)
        {
            List<T> chunk = list;
            list = new List<T>(size);
            yield return chunk;
        }
    }

    if (list.Count > 0)
    {
        yield return list;
    }
}

Usage:

var list = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };

foreach (var chunk in list.TakeChunks(3))
{
    Console.WriteLine(string.Join(", ", chunk));
}

Output:

1, 2, 3
4, 5, 6
7, 8, 9
10

Rationale:

Compared to other methods such multiple calls to Skip and Take or a big fancy LINQ query, the above is:

More efficient
More obvious in function (in my opinion)
More readable in implementation (again, in my opinion)

If you're going to return a copy each time, why not just start a new list for each batch and return that? Where's the benefit of calling ToArray? — Jon Skeet, Mar 02 '11 at 19:40
@Jon: Good question. I guess there wasn't really any good reason for that. — Dan Tao, Mar 02 '11 at 19:50

Donut · Answer 3 · 2011-03-02T19:41:54.973

4

One easy possibility is to use the Enumerable.Skip and Enumerable.Take methods, for example:

List<int> nums = new List<int>(){1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18};

var list1 = nums.Take(5);
var list2 = nums.Skip(5).Take(5);
var list3 = nums.Skip(10).Take(5);
var list4 = nums.Skip(15).Take(5);

As Jon mentioned in the comments though, a simple approach like this one will re-evaluate nums (in this example) each time, which will impact performance (depending on the size of the collection).

edited Mar 02 '11 at 19:41

answered Mar 02 '11 at 19:30

Donut

110,061
20
134
146

5

Note that this will re-evaluate the source each time - which is fine in some cases (e.g. in-memory collections) but not all (e.g. reading entries from large log files). Where possible, I try to write my LINQ code to only evaluate the source once. – Jon Skeet Mar 02 '11 at 19:35

Jon Skeet · Answer 4 · 2014-10-17T15:33:48.193

1

We have a Batch method in MoreLINQ. You need to be careful how you use it, as the batch that is passed to the selector each time is a reference to the same array - but it does work.

You can use GroupBy, but that can't be lazy - it has to accumulate all the results before it can return anything. That may be okay for you, but it's worth being aware of.

edited Oct 17 '14 at 15:33

answered Mar 02 '11 at 19:33

Jon Skeet

1,421,763
867
9,128
9,194

`Line 86: bucket = null;` Doesn't this ensure that the array is a different one each time? – CodesInChaos Mar 02 '11 at 20:18
@CodeInChaos: The `x => x` doesn't cause one allocation per batch - it's going to mean that each result is the same array reference (until the end). It's actually somewhat dangerous (e.g. `x.Batch(10).ToList()`) – Jon Skeet Mar 02 '11 at 20:18
@Jon I deleted my original comments because I missed the `bucket=null` at first. – CodesInChaos Mar 02 '11 at 20:19
@CodeInChaos: Oops, I hadn't spotted that myself. I'm not at all sure what we're doing there... it seems a little odd to me. – Jon Skeet Mar 02 '11 at 20:22
As I understand it the `bucket = null;` makes sure each batch has its own bucket, and thus the `ToList()` code works correctly. And the `Select(x=>x)` hides the array from the outside, but causes one(or two?) additional allocation and loses `IList`. So I still think wrapping in `ReadOnlyCollection` instead of `Select(x=>x)` might be a good idea. – CodesInChaos Mar 02 '11 at 20:25
@CodeInChaos: Ah, that `x => x` - sorry, I was looking at a different one. Yes, I'm not at all sure about that. To be honest, Batch wasn't one of my classes to start with... I may want to revisit this implementation... – Jon Skeet Mar 02 '11 at 20:29
1

The link to the `batch` function isn't working anymore. – Philipp M Oct 17 '14 at 15:19

score 0 · Answer 5 · answered Mar 02 '11 at 19:44

0

var list = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 };
var result = new List<List<int>>();
while (list.Count != 0) {
    result.Add(list.TakeWhile(x => x++ <= 5).ToList());
    list.RemoveRange(0, list.Count < 5 ? list.Count : 5);
}

answered Mar 02 '11 at 19:44

Dennis Traub

50,557
7
93
108

Divide a large IEnumerable into smaller IEnumerable of a fix amount of item

5 Answers5

Linked