Using Polly with async method does not continue with await

Question

I have the following method that makes a call to an async method to get an open MailKit connection:

    public async Task Process(string[] userEmails, IEmailMessage emailMessage) {
        var smtpClient = await _smtpClient.GetOpenConnection();

        _logger.Information("Breaking users into groups and sending out e-mails");
        // More Code
    }

The method GetOpenConnection looks like this:

    public async Task<IMailTransport> GetOpenConnection() {
        var policy = Policy.Handle<Exception>()
            .WaitAndRetryForeverAsync(_ => TimeSpan.FromSeconds(60),
                (exception, _) => _logger.Error(exception, "Could not establish connection"));

        return await policy.ExecuteAsync(EstablishConnection);
    }

    private async Task<IMailTransport> EstablishConnection() {
        var smtpConfiguration = _smtpConfiguration.GetConfiguration();

        _logger.Information("Get an open connection for SMTP client");

        _smtpClient.Connect(smtpConfiguration.Network.Host, smtpConfiguration.Network.Port,
            SecureSocketOptions.None, CancellationToken.None);
        var smtpClient = _smtpClient.GetSmtpClient();

        return await Task.FromResult(smtpClient);
    }

I turned off my test SMTP server (smtp4dev) and getting the connection fails as expected. Polly dutifully retries getting the connection every 60 seconds. However, when I turn the test SMTP server back on, the code does not continue where it was awaited in the Process method and I cannot figure out what I'm doing wrong.

If I convert GetOpenConnection to a synchronous method everything works properly, but, obviously, code execution is blocked until the connection is returned.

Any assistance is appreciated.

Update:

One additional item of note, is that the code executes properly if there is no error in getting the connection. If we can successfully grab the connection the first time, it will then move to the _logger line in the Process method and execute the rest of the code. It fails to move to the _logger line when we fail to grab a connection and have to retry execution.

Update 2:

Changed the Connect method to the following:

    public async Task Connect(string host, int port = 0, SecureSocketOptions options = SecureSocketOptions.Auto,
        CancellationToken cancellationToken = default) {
        await _smtpClient.ConnectAsync(host, port, options, cancellationToken);
    }

Updated the EstablishConnection to:

    private async Task<IMailTransport> EstablishConnection() {
        var smtpConfiguration = _smtpConfiguration.GetConfiguration();

        _logger.Information("Get an open connection for SMTP client");

        await _smtpClient.Connect(smtpConfiguration.Network.Host, smtpConfiguration.Network.Port,
            SecureSocketOptions.None, CancellationToken.None);

        var smtpClient = _smtpClient.GetSmtpClient();

        _logger.Information("Got an open connection for SMTP client");

        return await Task.FromResult(smtpClient);
    }

When I do this, now Polly does not appear to retry the connection at all.

Update 3:

    public void OnPostInsert(PostInsertEvent @event) {
        _logger.Information("OnPostInsert");

        _ = DispatchEvents(@event.Entity)
            .ContinueWith(x => _logger.Error(x.Exception,
                    "An error occurred while dispatching the insert event"),
                TaskContinuationOptions.OnlyOnFaulted);
    }

    private async Task DispatchEvents(object domainEntity) {
        Ensure.That(domainEntity, nameof(domainEntity)).IsNotNull();

        _logger.Information("Dispatch domain events");

        var entityBaseType = domainEntity.GetType().BaseType;

        if (entityBaseType is not { IsGenericType: true }) return;

        if (entityBaseType.GetGenericTypeDefinition() != typeof(EntityBase<>))
            return;

        if (domainEntity.GetType().GetProperty("DomainEvents")?.GetValue(domainEntity, null) is not
            IReadOnlyList<INotification> domainEvents) { return; }

        foreach (var domainEvent in domainEvents) {
            _logger.Information("Publishing Event {@DomainEvent}", domainEvent);
            await _mediator.Publish(domainEvent, CancellationToken.None);
        }
    }

Answer 1

The problem was not MailKit or Polly. As one of the commenters above indicated and also through my own testing, Polly wasn't the issue and worked as advertised. I downloaded MailKit, compiled a debug version, ripped out the release version, and added references to the new library.

After debugging, I found MailKit also worked and established a connection. While debugging, I had a random thought that maybe the problem was leaving off ConfigureAwait(false) from my async calls and, sure enough, it was.

In a sense, I created the problem for myself because I deliberately left this off of the asynchronous calls. I experienced issues with Serilog where I found I lost the ExecutionContext in async calls and, as a result, Serilog could not capture the CorrelationId and other properties added to the original thread by enrichers. Because the ExecutionContext wasn't transferred, the SecurityContext also was not transferred so bye-bye UserPrincipal. Removing ConfigureAwait(false) seemed to remove this issue.

Once I added ConfigureAwait(false) to the async calls, magic started happening. Honestly, I don't have a complete understanding of async to posit why the removal of this method caused these issues. I understand that ConfigureAwait(false) can improve performance because there is a cost to queuing a callback to synchronize the contexts and to potentially avoid any deadlocks created through this synchronization.

Now that I have nailed down the source of the problem, I just need to figure out how to transfer the ExecutionContext (SecurityContext, etc.) to flow down the async calls.

Process:

    public async Task Process(string[] userEmails, IEmailMessage emailMessage) {
        var smtpClient = await _smtpClient.GetOpenConnection().ConfigureAwait(false);

        _logger.Information("Breaking users into groups and sending out e-mails");

        // ReSharper disable once ForCanBeConvertedToForeach
        foreach (var addressGroup in userEmails.BreakArrays(_configuration.EmailGroupSize.Size))
            await _sender.SendEmailMessage(addressGroup, smtpClient, emailMessage);

        _logger.Information("Emails sent");

        await smtpClient.DisconnectAsync(true);
        smtpClient.Dispose();
    }

GetOpenConnection:

    public async Task<IMailTransport> GetOpenConnection() {
        var smtpConfiguration = _smtpConfiguration.GetConfiguration();

        var policy = Policy.Handle<Exception>()
            .WaitAndRetryForeverAsync(_ => TimeSpan.FromSeconds(60),
                (exception, _) => _logger.Error(exception, "Could not establish connection"));

        await policy.ExecuteAsync(async () => await _smtpClientAdapter.ConnectAsync(smtpConfiguration.Network.Host,
            smtpConfiguration.Network.Port,
            SecureSocketOptions.None, CancellationToken.None).ConfigureAwait(false)).ConfigureAwait(false);

        return await Task.FromResult(_smtpClientAdapter.GetSmtpClient());
    }

ConnectAsync on the MailKitSmtpClientAdapter class:

    public async Task ConnectAsync(string host, int port = 0, SecureSocketOptions options = SecureSocketOptions.Auto,
        CancellationToken cancellationToken = default) {
        await _smtpClient.ConnectAsync(host, port, options, cancellationToken).ConfigureAwait(false);
    }