How to split an array into chunks of specific size?

Question

Afternoon,

I need to split an array into smaller "chunks".

I am passing over about 1200 items, and need to split these into easier to handle arrays of 100 items each, which I then need to process.

Could anyone please make some suggestions?

Answer 1

Array.Copy has been around since 1.1 and does an excellent job of chunking arrays.

List.GetRange() would also be a good choice as mentioned in another answer.

string[] buffer;

for(int i = 0; i < source.Length; i+=100)
{
    buffer = new string[100];
    Array.Copy(source, i, buffer, 0, 100);
    // process array
}

And to make an extension for it:

public static class Extensions
{
    public static T[] Slice<T>(this T[] source, int index, int length)
    {       
        T[] slice = new T[length];
        Array.Copy(source, index, slice, 0, length);
        return slice;
    }
}

And to use the extension:

string[] source = new string[] { 1200 items here };

// get the first 100
string[] slice = source.Slice(0, 100);

Update: I think you might be wanting ArraySegment<> No need for performance checks, because it simply uses the original array as its source and maintains an Offset and Count property to determine the 'segment'. Unfortunately, there isn't a way to retrieve JUST the segment as an array, so some folks have written wrappers for it, like here: ArraySegment - Returning the actual segment C#

ArraySegment<string> segment;
           
for (int i = 0; i < source.Length; i += 100)
{
    segment = new ArraySegment<string>(source, i, 100);

    // and to loop through the segment
    for (int s = segment.Offset; s < segment.Array.Length; s++)
    {
        Console.WriteLine(segment.Array[s]);
    }
}

---

Performance of Array.Copy vs Skip/Take vs LINQ

Test method (in Release mode):

static void Main(string[] args)
{
    string[] source = new string[1000000];
    for (int i = 0; i < source.Length; i++)
    {
        source[i] = "string " + i.ToString();
    }

    string[] buffer;

    Console.WriteLine("Starting stop watch");

    Stopwatch sw = new Stopwatch();

    for (int n = 0; n < 5; n++)
    {
        sw.Reset();
        sw.Start();
        for (int i = 0; i < source.Length; i += 100)
        {
            buffer = new string[100];
            Array.Copy(source, i, buffer, 0, 100);
        }

        sw.Stop();
        Console.WriteLine("Array.Copy: " + sw.ElapsedMilliseconds.ToString());

        sw.Reset();
        sw.Start();
        for (int i = 0; i < source.Length; i += 100)
        {
            buffer = new string[100];
            buffer = source.Skip(i).Take(100).ToArray();
        }
        sw.Stop();
        Console.WriteLine("Skip/Take: " + sw.ElapsedMilliseconds.ToString());

        sw.Reset();
        sw.Start();
        String[][] chunks = source                            
            .Select((s, i) => new { Value = s, Index = i })                            
            .GroupBy(x => x.Index / 100)                            
            .Select(grp => grp.Select(x => x.Value).ToArray())                            
            .ToArray();
        sw.Stop();
        Console.WriteLine("LINQ: " + sw.ElapsedMilliseconds.ToString());
    }
    Console.ReadLine();
}

---

Results (in milliseconds):

Array.Copy:    15
Skip/Take:  42464
LINQ:         881

Array.Copy:    21
Skip/Take:  42284
LINQ:         585

Array.Copy:    11
Skip/Take:  43223
LINQ:         760

Array.Copy:     9
Skip/Take:  42842
LINQ:         525

Array.Copy:    24
Skip/Take:  43134
LINQ:         638

Answer 2

You can use LINQ to group all items by the chunk size and create new Arrays afterwards.

// build sample data with 1200 Strings
string[] items = Enumerable.Range(1, 1200).Select(i => "Item" + i).ToArray();
// split on groups with each 100 items
String[][] chunks = items
                    .Select((s, i) => new { Value = s, Index = i })
                    .GroupBy(x => x.Index / 100)
                    .Select(grp => grp.Select(x => x.Value).ToArray())
                    .ToArray();

for (int i = 0; i < chunks.Length; i++)
{
    foreach (var item in chunks[i])
        Console.WriteLine("chunk:{0} {1}", i, item);
}

Note that it's not necessary to create new arrays(needs cpu cycles and memory). You could also use the IEnumerable<IEnumerable<String>> when you omit the two ToArrays.

Here's the running code: http://ideone.com/K7Hn2

Answer 3

here I found another linq-solution:

int[] source = new[] { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
int i = 0;
int chunkSize = 3;
int[][] result = source.GroupBy(s => i++ / chunkSize).Select(g => g.ToArray()).ToArray();

//result = [1,2,3][4,5,6][7,8,9]

Answer 4

You can use Skip() and Take()

string[] items = new string[]{ "a", "b", "c"};
string[] chunk = items.Skip(1).Take(1).ToArray();

Answer 5

    string[]  amzProductAsins = GetProductAsin();;
    List<string[]> chunks = new List<string[]>();
    for (int i = 0; i < amzProductAsins.Count; i += 100)
    {
        chunks.Add(amzProductAsins.Skip(i).Take(100).ToArray());
    }

Answer 6

You can use List.GetRange:

for(var i = 0; i < source.Count; i += chunkSize)
{
    List<string> items = source.GetRange(i, Math.Min(chunkSize, source.Count - i));
}

Although not at fast as Array.Copy, I think it looks cleaner:

var list = Enumerable.Range(0, 723748).ToList();

var stopwatch = new Stopwatch();

for (int n = 0; n < 5; n++)
{
    stopwatch.Reset();
    stopwatch.Start();
    for(int i = 0; i < list.Count; i += 100)
    {
        List<int> c = list.GetRange(i, Math.Min(100, list.Count - i));
    }
    stopwatch.Stop();
    Console.WriteLine("List<T>.GetRange: " + stopwatch.ElapsedMilliseconds.ToString());

    stopwatch.Reset();
    stopwatch.Start();
    for (int i = 0; i < list.Count; i += 100)
    {
        List<int> c = list.Skip(i).Take(100).ToList();
    }
    stopwatch.Stop();
    Console.WriteLine("Skip/Take: " + stopwatch.ElapsedMilliseconds.ToString());

    stopwatch.Reset();
    stopwatch.Start();
    var test = list.ToArray();
    for (int i = 0; i < list.Count; i += 100)
    {
        int length = Math.Min(100, list.Count - i);
        int[] c = new int[length];
        Array.Copy(test, i, c, 0, length);
    }
    stopwatch.Stop();
    Console.WriteLine("Array.Copy: " + stopwatch.ElapsedMilliseconds.ToString());

    stopwatch.Reset();
    stopwatch.Start();
    List<List<int>> chunks = list
        .Select((s, i) => new { Value = s, Index = i })
        .GroupBy(x => x.Index / 100)
        .Select(grp => grp.Select(x => x.Value).ToList())
        .ToList();
    stopwatch.Stop();
    Console.WriteLine("LINQ: " + stopwatch.ElapsedMilliseconds.ToString());
}

Results in milliseconds:

List<T>.GetRange: 1
Skip/Take: 9820
Array.Copy: 1
LINQ: 161

List<T>.GetRange: 9
Skip/Take: 9237
Array.Copy: 1
LINQ: 148

List<T>.GetRange: 5
Skip/Take: 9470
Array.Copy: 1
LINQ: 186

List<T>.GetRange: 0
Skip/Take: 9498
Array.Copy: 1
LINQ: 110

List<T>.GetRange: 8
Skip/Take: 9717
Array.Copy: 1
LINQ: 148

Answer 7

Use LINQ, you can use Take() and Skip() functions

Answer 8

General recursive extension method:

    public static IEnumerable<IEnumerable<T>> SplitList<T>(this IEnumerable<T> source, int maxPerList)
    {
        var enumerable = source as IList<T> ?? source.ToList();
        if (!enumerable.Any())
        {
            return new List<IEnumerable<T>>();
        }
        return (new List<IEnumerable<T>>() { enumerable.Take(maxPerList) }).Concat(enumerable.Skip(maxPerList).SplitList<T>(maxPerList));
    }

Answer 9

If you have an array to be divided but the division has the rest with this simple solution you can share the elements missing into the various "chunks" equally.

int[] arrInput = new int[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 };
var result = SplitArrey(arrInput, 5);
foreach (var item in result) {
  Console.WriteLine("   {0}", String.Join(" ", item));
}

the function is:

public static List<int[]> SplitArrey(int[] arrInput, int nColumn) {

        List<int[]> result = new List<int[]>(nColumn);
    
        int itemsForColum = arrInput.Length / nColumn;  
        int countSpareElement = arrInput.Length - (itemsForColum * nColumn);    

        // Add and extra space for the spare element
        int[] newColumLenght = new int[nColumn];
        for (int i = 0; i < nColumn; i++)
        {
            int addOne = (i < countSpareElement) ? 1 : 0;
            newColumLenght[i] = itemsForColum + addOne;
            result.Add(new int[itemsForColum + addOne]);
        }

        // Copy the values
        int offset = 0;
        for (int i = 0; i < nColumn; i++)
        {
            int count_items_to_copy = newColumLenght[i];
            Array.Copy(arrInput, offset, result[i], 0, count_items_to_copy);
            offset += newColumLenght[i];
        }
        return result;
    }

the result is:

1 2 3
4 5 6
7 8
9 10
11 12