How do lambdas scope to local variables?

Question

So my understanding of how the compiler handles lambdas is limited.

My understanding is that the compiler takes your lambda and turns it into a real method.

If that's the case then how does it scope to local variables?

    public async Task<dynamic> GetWebStuff()
    {
        dynamic ret = "";

        WebClient wc = new WebClient();          

        wc.DownloadStringCompleted += async (s, a) => 
        {
            ret = await Newtonsoft.Json.JsonConvert.DeserializeObject(a.Result.ToString());
        };

        wc.DownloadString("http://www.MyJson.com");

        return ret;
    }

The above example will set the return value of ret to the caller which is a dynamic object of deserialized JSON.

How does that happen though if the compiler takes that completed event lambda and abstracts it into its own method? How does it know to set the ret value?

It's like me saying this (which obviously wont work)

        public async Task<dynamic> GetWebStuff()
        {
            dynamic ret = "";

            WebClient wc = new WebClient();

            wc.DownloadStringCompleted += wc_DownloadStringCompleted;            

            wc.DownloadString("google.com");

            return ret;
        }

        void wc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
        {
            ret = await Newtonsoft.Json.JsonConvert.DeserializeObject(e.Result.ToString());
        }

Answer 1

It does that creating an anonymous class. For example consider this code:

int x = 0;

Action action = () => x = 2;

action();

Console.Write(x);

And the generated class :

IL code of the <Main>b__2 method which sets the value of x:

    .method assembly hidebysig instance void 
        '<Main>b__2'() cil managed
{
  // Code size       10 (0xa)
  .maxstack  8
  IL_0000:  ldarg.0
  IL_0001:  ldc.i4.2
  IL_0002:  stfld      int32 ConsoleApplication1.Program/'<>c__DisplayClass0'::x
  IL_0007:  br.s       IL_0009
  IL_0009:  ret
} // end of method '<>c__DisplayClass0'::'<Main>b__2'

Answer 2

I recommend not focusing on how the compiler does such a thing as that is an implementation detail that can change over time and as others have said different compiler implementations (mono?). Instead, know that such a thing happens because of closure.

In Wiki:

In programming languages, a closure (also lexical closure or function closure) is a function or reference to a function together with a referencing environment—a table storing a reference to each of the non-local variables (also called free variables or upvalues) of that function.1 A closure—unlike a plain function pointer—allows a function to access those non-local variables even when invoked outside its immediate lexical scope.

Answer 3

So first off, your code doesn't work. It doesn't work for reasons entirely unrelated to your question, but it doesn't work nonetheless.

The code does successfully mutate ret to be the result of the downloaded value, when the download actually completes. Sadly, you have already returned a value long before that happens because you return the result of the task without waiting for the download to finish.

You'll notice that your implementation of GetWebStuff actually generates a compiler warning stating that "This async method lacks 'await' operators and will run synchronously [...]". Whenever you see this warning you can be virtually certain that your method is not designed properly, because it appears to be asynchronous while not actually doing anything asynchronously.

Here is a valid implementation of your method:

public Task<dynamic> GetWebStuff()
{
    var tcs = new TaskCompletionSource<dynamic>();

    WebClient wc = new WebClient();

    wc.DownloadStringCompleted += async (s, a) =>
    {
        tcs.TrySetResult(await Newtonsoft.Json.JsonConvert.DeserializeObject(
            a.Result.ToString()));
    };

    wc.DownloadStringAsync(new Uri("http://www.MyJson.com"));

    return tcs.Task;
}

Here the method returns a task that won't be completed until the event is fired, at which point the result of the task is set to be the result of the download.

As for how the closure works, it's easiest to simply see what the compiler transforms this code into:

class ClosureClass
{
    public TaskCompletionSource<dynamic> tcs;
    public async Task AnonymousMethod1(object s, 
        DownloadDataCompletedEventArgs a)
    {
        tcs.TrySetResult(await Newtonsoft.Json.JsonConvert.DeserializeObject(
            a.Result.ToString()));
    }
}

public Task<dynamic> GetWebStuff()
{
    ClosureClass closure = new ClosureClass();
    closure.tcs = new TaskCompletionSource<dynamic>();

    WebClient wc = new WebClient();

    wc.DownloadStringCompleted += closure.AnonymousMethod1;

    wc.DownloadStringAsync(new Uri("http://www.MyJson.com"));

    return closure.tcs.Task;
}

When closing over a variable a new closure class is created, with an instance variable for each closed over variable. The lambda is turned into an instance method that uses those instance fields. The method that had the lambda creates an instance of that new type, uses its fields rather than locals wherever those closed over locals were used, and then uses the new named method where the lambda was.