Splitting a List into chunks - C#

Question

I have a list which consists of around 1,000,000 records. And, I am using EF to insert it into the database. Since it takes too long I need to split this List into blocks of 100 records and save.

How can I do this?

await dbCont.People.AddAsync(peoples);
await dbCont.Savechanges();

The people's List here contains 1,000,000 records. And instead of saving all at once (slows the insert), I need to insert around 100 records at a time. How can I do this?

Have a look at linq `Skip` and `Take`: https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/linq/return-or-skip-elements-in-a-sequence — Tom John, Apr 23 '19 at 10:10
@TomJohn Are you aware of any method where I could insert bulk using EF ? — Illep, Apr 23 '19 at 10:10
You're aware that this will likely create 10.000 transactions, if one of them fails, the previous transactions have already been committed? — Lasse V. Karlsen, Apr 23 '19 at 10:11
@LasseVågsætherKarlsen How do you think I could overcome my issue ? — Illep, Apr 23 '19 at 10:12
Take a look at this thread https://stackoverflow.com/questions/11463734/split-a-list-into-smaller-lists-of-n-size - it might help. — nelion, Apr 23 '19 at 10:17
Keep the code as-is, or try to follow up on true bulk insert solution. Problem is that entity framework is doing a lot of work behind the scenes and bulk insert might bypass that so it may not be possible to use. — Lasse V. Karlsen, Apr 23 '19 at 10:18
@LasseVågsætherKarlsen Is there any Nuget packages/framework you are aware that I could use? — Illep, Apr 23 '19 at 10:27
A big performance issue is that during EF generates an INSERT for each object when calling SaveChanges. There is a bulk insert extensibility in the EF Core library, but it's not cheap. — Metheny, Apr 23 '19 at 10:39
Does this answer your question? [Split a List into smaller lists of N size](https://stackoverflow.com/questions/11463734/split-a-list-into-smaller-lists-of-n-size) — Drag and Drop, Mar 23 '20 at 12:29
MyList.Chunk(100) is also now available in .Net 7 https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.chunk?view=net-7.0 — Rob, Jul 04 '23 at 09:59

score 1 · Answer 1 · answered Apr 23 '19 at 10:19

You can do it like this:

var batchSize = 100;
var processed = 0;
var hasNextBatch = true;

while(hasNextBatch) 
{
    var batch = peoples.Skip(processed).Take(batch).ToList();
    await dbCont.People.AddAsync(batch);
    await dbCont.Savechanges();
    processed += batch.count;
    hasNextBatch = batch.Count == batchSize;
}

but you will have to consider that when a batch insert fails, the previous batches are already committed.

score 1 · Answer 2 · answered Apr 23 '19 at 10:27

You could use the extension method Batch of the library MoreLinq: Batches the source sequence into sized buckets.

var list = new List<int>(Enumerable.Range(1, 25));
var buckets = list.Batch(size: 10);
foreach (var bucket in buckets)
{
    Console.WriteLine(String.Join(", ", bucket));
}

Output

1, 2, 3, 4, 5, 6, 7, 8, 9, 10
11, 12, 13, 14, 15, 16, 17, 18, 19, 20
21, 22, 23, 24, 25

score 0 · Answer 3 · answered Apr 23 '19 at 10:39

0

I have tried to insert chunks in EF in past, but it is slow because of entity tracking. Of course you can disable entity tracking, but better use SqlBulkCopy to bulk insert. Here is a good article Bulk inserting data into SQL Server.

answered Apr 23 '19 at 10:39

nikolai.serdiuk

762
8
11

Can I use something that goes along with EF ? – Illep Apr 23 '19 at 10:49
EF is bad for bulk insert, in case you want bulk insert using entities you can use Dapper: https://dapper-tutorial.net/bulk-insert – nikolai.serdiuk Apr 24 '19 at 11:38

score 0 · Answer 4 · answered Jul 04 '23 at 09:49

Below is to split in 500 chunks, we can use any number to chunk of 100 or 1000, instead of 500. Just replace and get going. Happy Coding!!

C# Code:

 public string OrdersListsplitAndProcess(List<OrderInfo> OrigOrdersList)
        {
            int rowscount = 0, totCount = OrigOrdersList.Count;
            int remainder = (totCount % 500); int FinalQuotient = Convert.ToInt32(Math.Floor(Convert.ToDecimal(totCount / 500)));
            int thisQuotient = 0;
            List<OrderInfo> thisOrdersList = new List<OrderInfo>();
            for (int i = 0; i < totCount; i++)
            {
                thisOrdersList.Add(OrigOrdersList[i]);
                rowscount++;
                if (rowscount == 500 || (thisQuotient == FinalQuotient && rowscount == remainder))
                {
                    processOpenOrders(thisOrdersList);// CHUNCKED LIST
                    thisOrdersList.Clear(); // TO RESET and PREPARE FOR NEXT CHUNK
                    if (rowscount != remainder)
                        thisQuotient++;
                    rowscount = 0;
                }
            }
            return "finished successfully";
        } ```

Splitting a List into chunks - C#

4 Answers4