Entity Framework context retaining local data after being re-created

Question

I am trying to do large inserts using entity framework, so to reduce context bloat I am trying to recreate the context every x loops. However, after recreating the context the data from the previous context is still in the local information. I have tried calling Dispose() on the context and setting it to null before recreating it.

I found this question asked before where someone seems to have the same issue, but there was no solution found. Entity Framework Context - Recreating Object keeps old object in memory

ApplicationDbContext context = new ApplicationDbContext();
context.Configuration.AutoDetectChangesEnabled = false;

for (int i = 0; i < products.Count; i++)
 {
    context.Product.Add(products[i]);
    if(i % 50 == 0)
    {
        context.SaveChanges();
        context.Dispose();
        context = null;
        context = new ApplicationDbContext();
        context.Configuration.AutoDetectChangesEnabled = false;
    }
}
context.SaveChanges();

Context Product local

For clarification, this is what I mean when the local data is not cleared. The local data for the Products has 151 entries locally, despite having the context be recreated every 50 entries. This has lead to the products being added multiple times to the database.

Just to clarify what I am asking, how can I recreate the context so that it does not have an data left over from the previous context? Also, if anyone could explain what causes the data to be retained to begin with that would be nice to understand.

Edit: I have tried restructuring the code as suggested by by Gert Arnold in the comments. Here is the current version.

        public void BatchAddProducts(List<ProductModel> products)
        {

            int loops = (int)Math.Ceiling((double)(products.Count / 50));
            List<ProductModel> sublist = new List<ProductModel>();
            for (int i = 0; i < loops; i++)
            {
                int toPull;
                if((i * 50 + 50) > products.Count)
                {
                    toPull = products.Count - (i * 50);
                }
                else
                {
                    toPull = 50;
                }
                sublist = products.GetRange(i * 50, toPull);
                ProductAdd(sublist);
            }
        }

        private void ProductAdd(List<ProductModel> products)
        {
            using(ApplicationDbContext context = new ApplicationDbContext())
            {
                context.Product.Local.Clear();
                context.Product.AddRange(products);
                context.SaveChanges();
            }
        }

This still has the same issue. The context is retaining information from the version that should no longer exist, and entries that were added in the previous loop are being added to the database again. i.e. after 2 iterations 150 entries are added to the database, and after 3 there are 300 new entries in the database.

In response to the answer by Steve Py, I am checking the context.Product.Local information just before SaveChanges() is called. The duplicate data I mention is duplicates of the products that are in the context. So to refer to the example I made earlier, after 3 loops, there are 3 versions of a product from the 1st iteration, 2 from the 2nd, and 1 from the 3rd. I dont believe it is an issue with how the product list is being generated. That was my first thought when I encountered this, and I checked that code. This code is being written as an upgrade to improve large inserts, and the code that generates the products is older that worked fine with the old update system. Also, the list being passed to the function only has ~8000 Products, so it cant be that as far as I can tell.

Answer 1

I suspect that you may be encountering a different issue based on the statement "this has lead to the products being added multiple times to the database" and your process of verifying the local cache is resulting in the DBSet being loaded into memory.

I have tested the disposed DbContext behaviour around the .Local and it is behaving as expected, the previously loaded entities are not still referenced.

    [Test]
    public void TestLocalReset()
    {
        var context = new TestDbContext();

        for (int count = 1; count <= 100; count++)
        {
            var note = new Note { Text = $"Note #{count}." };
            context.Notes.Add(note);
            context.SaveChanges();
            if (count % 10 == 0)
            {
                int test = context.Notes.Local.Count; // <- Always 10
                context.Dispose();
                context = new TestDbContext();
                test = context.Notes.Local.Count; // <- Always 0
            }
        }
        context.Dispose();
    }

Your example does not show how/where you are checking the .Local instance, but if you have any code that queries against your DbSet that could result in loading the entire results into memory and showing up in the .Local instance.

As for duplicate data, are you getting duplicate Products, or do you have other referenced entities beneath the Product that you are seeing duplicated? How are you building/populating these Products in your array? References that might have been loaded by the outermost DbContext to be referenced by these new products will not be tracked when you recreate the DbContext resulting in potentially duplicate related entities being inserted.

You may need to update your question to show the complete code you are using to create the Products and insert them as this will likely contain details explaining the behaviour you are seeing.