Why can't property bag (Dictionary) be modified inside IEnumerable loop

Question

Consider the following test program in which I (ab)use a dictionary to contain a document which may have unknown fields (and unknown types for those fields) ,

Fiddle of the program

using System;
using System.Linq;
using System.Collections.Generic;
using Newtonsoft.Json;

public class Program
{
    public static void Main()
    {
        var docs = GetDocuments();
        foreach(var doc in docs){
            doc["a"] = new string[]{"Hello", "World!"};
            var docInLoop = JsonConvert.SerializeObject(doc);
            Console.WriteLine(docInLoop);
        }

        var serialized = JsonConvert.SerializeObject(docs);
        Console.WriteLine("===========================================================================================");
        Console.WriteLine(serialized);
        Console.WriteLine("===========================================================================================");
        var bar = docs.First()["a"] as string[];        
        Console.Write("First entry of first document is string[]?");
        Console.WriteLine(bar==null? " No" : "Yes");


    }

    public static IEnumerable<Document> GetDocuments(){
        return Enumerable.Range(0, 10).Select(i => {

        var doc = new Document();
        doc["a"] = new int[]{1,2,3,4,5,6};
        return doc;
        });
    }

    public class Document : Dictionary<string, object>{}
}

When running this, the expectation is that since in the foreach loop I modify the document the collection of documents should be modified. But here is the output:

{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
{"a":["Hello","World!"]}
===========================================================================================
[{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]},{"a":[1,2,3,4,5,6]}]
===========================================================================================
First entry of first document is string[]? No

Judging by the deserialization of the collection, mutating the document in the loop has no effect? How is this possible? What am I missing? I have a direct reference to the document object in the loop...

This looks to me like it could be a problem related to the deferred execution behaviour of `IEnumerable`. Have you tried creating an actual collection of the documents? (i.e. something like `var docs = GetDocuments().ToArray()`) — bassfader, Jul 24 '19 at 12:28
I am expecting it to modify the dictionary, which is what I am doing inside the loop. It does not, I should be able to assign any instanec of a reference type to any key in that dictionary (including re-assigning to a new reference type). — arynaq, Jul 24 '19 at 12:30
@bassfader Seems like you are right... forcing it to evaluate fixes it, but then again I would expect newtonsoft to evaluate the IEnumerable — arynaq, Jul 24 '19 at 12:32
`but then again I would expect newtonsoft to evaluate the IEnumerable` It does evaluate the enumerable. And thus gets **a set of completely new `Document`s**. Note - this is nothing at all to do with newtonsoft - you'd see **exactly the same thing** if you did a second `foreach` loop. — mjwills, Jul 24 '19 at 12:33

Mong Zhu · Accepted Answer · 2019-07-24T12:39:59.050

I have a direct reference to the document object in the loop...

No you don't. You have the reference to the recipe of how to cook up this data! This is what deferred execution is about.

Actually the query is executed in this line the first time:

foreach(var doc in docs){

Here you cooked up the meal for the first time. You spiced it nicely with your own ingredients.

When you serialize doc in this line:

var serialized = JsonConvert.SerializeObject(docs);

you are basically executing the query again that is in GetDocuments. It is the same as if you would write:

var serialized = JsonConvert.SerializeObject(GetDocuments());

That means basically that you cook the meal again. Following the recipe, but this time you don't add any ingredients as the last time. And then you wonder why the soup does not taste like the spices you put in the first time.

If you materialize the result using a ToList() call before the loop you would get the desired result:

var docs = GetDocuments().ToList();

Here is a nice article elaborating on the traps of deferred execution

Thanks for the great explanation, I understand it better now. I just don't like the hidden implementation, reminds me of struggles with shallow copies in Qt :p — arynaq, Jul 24 '19 at 12:49
@arynaq you are welcome. Yeah shallow copies have also nice traps. But these here are worse ;) — Mong Zhu, Jul 24 '19 at 12:56

Why can't property bag (Dictionary) be modified inside IEnumerable loop

1 Answers1