Process a list with a loop, taking 100 elements each time and automatically less than 100 at the end of the list

Question

Is there a way to use a loop that takes the first 100 items in a big list, does something with them, then the next 100 etc but when it is nearing the end it automatically shortens the "100" step to the items remaining.

Currently I have to use two if loops:

for (int i = 0; i < listLength; i = i + 100)
{
    if (i + 100 < listLength)
    {
        //Does its thing with a bigList.GetRange(i, 100)
    }
    else
    {
        //Does the same thing with bigList.GetRange(i, listLength - i)
    }
}

Is there a better way of doing this? If not I will at least make the "thing" a function so the code does not have to be copied twice.

possible duplicate of [Split List into Sublists with LINQ](http://stackoverflow.com/questions/419019/split-list-into-sublists-with-linq) - this chunkifying behavior is exactly what is asked for, just with LINQ not only with plain loops&variables. So, maybe not an exact duplicate, but still worth considering and worth reading. — quetzalcoatl, Jun 07 '15 at 14:21

adricadar · Accepted Answer · 2015-06-07T15:15:58.377

129

You can make use of LINQ Skip and Take and your code will be cleaner.

for (int i = 0; i < listLength; i=i+100)
{
    var items = bigList.Skip(i).Take(100); 
    // Do something with 100 or remaining items
}

Note: If the items are less than 100 Take would give you the remaining ones.

edited Jun 07 '15 at 15:15

answered Jun 07 '15 at 14:17

adricadar

9,971
5
33
46

2

Thanks, works like a charm. Also very clear! I needed a string of the list items thus it became: `string.Join(",", idList.Skip(i).Take(100))` – SecondLemon Jun 07 '15 at 14:38
what if i don't know the listlength and there is no way to get the count of the list items? – Shekhar Reddy Mar 15 '18 at 11:37
1

@ShekharReddy in C# there are 2 way to get the `var listLength = bigList.Count`, you can find more info on [this answer](https://stackoverflow.com/questions/4098186/lists-count-vs-count) – adricadar Mar 15 '18 at 12:14
Awesome, thats exactly what I was looking for. Clean and Simple +1 – Hamza Khanzada Jan 25 '20 at 11:53

score 17 · Answer 2 · answered Oct 18 '18 at 17:19

17

I didn't like any of the answers listed, so I made my own extension:

public static class IEnumerableExtensions
{
    public static IEnumerable<IEnumerable<T>> MakeGroupsOf<T>(this IEnumerable<T> source, int count)
    {
        var grouping = new List<T>();
        foreach (var item in source)
        {
            grouping.Add(item);
            if(grouping.Count == count)
            {
                yield return grouping;
                grouping = new List<T>();
            }
        }

        if (grouping.Count != 0)
        {
            yield return grouping;
        }
    }
}

Then you can use it:

foreach(var group in allItems.MakeGroupsOf(100))
{
    // Do something
}

answered Oct 18 '18 at 17:19

Kyle W

3,702
20
32

1

This is exactly what I was looking for too. Skip+Take to me seems like you're starting over at element 0 every time, while this is literally iterating the list only once, and yielding the bunches. – Josh Sutterfield Jan 29 '20 at 11:19
This is the right answer, although I'd sway away from using `grouping` names since [`Grouping`s](https://github.com/microsoft/referencesource/blob/master/System.Core/System/Linq/Enumerable.cs#L2298) are already a concept. It may cause confusion. Not to mention that `group` is a keyword in some versions of c# and in your second snippet a compiler error may be caused. – William Herrmann Feb 13 '21 at 14:06

score 7 · Answer 3 · answered Mar 16 '22 at 07:53

In dotnet 6 you can use chunk:

//Child collections will be comprised of 10 elements each.
IEnumerable<int[]> sublists = numbers.Chunk(10);

https://exceptionnotfound.net/bite-size-dotnet-6-chunk-in-linq/

There is also a reference to use a group by to do this, which is quite an interesting solution for older versions of the framework: Split a collection into `n` parts with LINQ?

score 6 · Answer 4 · answered Jun 07 '15 at 14:16

6

You can keep an explicit variable for the end point:

for (int i = 0, j; i < listLength; i = j)
{
    j = Math.min(listLength, i + 100);
    // do your thing with bigList.GetRange(i, j)
}

answered Jun 07 '15 at 14:16

Ted Hopp

232,168
48
399
521

score 2 · Answer 5 · edited Jun 07 '15 at 14:32

2

List<int> list = null;
int amount_of_hundreds = Math.Floor(list.Count/100);
int remaining_number = list.Count - (amount_of_hundreds * 100);

for(int i = 0; i < amount_of_hundreds; ++i)
    {
    for(int j = 0; j < 100; ++j)
        {
        int item = list[(i * 100) + j];
        // do what you want with item
        }
    }

 for(int i = 0; i < remaining_number; ++i)
    {
    int item = list[(amount_of_hundreds * 100) + i];
    // do what you want with item
    }

edited Jun 07 '15 at 14:32

quetzalcoatl

32,194
8
68
107

answered Jun 07 '15 at 14:19

user3476093

704
2
6
11

@quetzalcoatl - amended 0 to i, but what do you mean by manual and duplicated? – user3476093 Jun 07 '15 at 14:32
mostly just the three: "// do what you want with item" is required to be written twice (yeah, small thing, but newbies will not think of a method and will copy-paste bodies), main and finishing loops are separate (what for? nothing seems gained. could squash them into one with `Math.min`), "amount_of_hundreds; remaining_number" could use div-mod instead of calculating the 'remaining' part manually. Just cosmetics. – quetzalcoatl Jun 07 '15 at 14:35

score 1 · Answer 6 · answered Nov 10 '20 at 12:47

1

You can split one big List to many small lists by limit, like this:

var limit = 100;
foreach (var smallList in bigList.Select((x, i) => new { Index = i, Value = x })
                                 .GroupBy(x => x.Index / limit)
                                 .Select(x => x.Select(v => v.Value).ToList())
                                 .ToList())
{
    Console.WriteLine($"{smallList.Count}");
    Console.WriteLine(String.Join("\r\n", smallList));
}

answered Nov 10 '20 at 12:47

Geograph

2,274
23
23

Just what i needed but i dont understand why it works. My linq is not good enough to understand this part: "((x, i) => new { Index = i, Value = x })" – Jipo Mar 27 '23 at 03:27
1

@Jipo That `.Select((x, i))` syntax is used to select both the value and the index of that value from the list. The `new { Index = i, Value = x }` afterwards is simply creating an anonymous type and naming the index and value appropriately. – Loz Apr 05 '23 at 15:18

nabanita · Answer 7 · 2019-04-16T06:26:46.270

you can try below approach also:

        int i = 1;
        foreach (item x in bigList)
        {

            batchOperation.Insert(x); //replace it with your action; create the batch
            i++;
            if (i >100)
            {
                table.ExecuteBatch(batchOperation); //execute the batch
                batchOperation.Clear();
                i = 1; // re-initialize 
            }

        }

        if (batchOperation.Count >= 1 )
        {
            table.ExecuteBatch(batchOperation); //do this for the residue items 
        }

score 0 · Answer 8 · answered Apr 16 '20 at 09:56

This is my solution, before all, i skip the last rows (rows - skiplast) after i get top of the array.

    [TestMethod]
    public void SkipTake()
    {
        var rows = 100;
        var skip = 1;
        var skiplast = 95;
        var result = Enumerable.Range(1, rows).Take(rows - skiplast).Skip(skip - 1 == 0 ? 1 : skip).ToList();
    }

Process a list with a loop, taking 100 elements each time and automatically less than 100 at the end of the list

8 Answers8