WebApi 2.2 conditional middleware or another way to encrypt response of specific controllers

Question

I've build very simple middleware that encrypts response, it works fine. Because this middleware is added as first inside Configuration in Startup.cs is applies to all requests. I'd like to be able to encrypt only responses from certain controllers and/or methods, ideally by adding attribute to them.
I've searched a bit and I found that it is impossible to get controller and method inside middleware - Web Api 2 get controller and action name in OWIN middleware?

My question is how can I create such functionality? What are my other options?
I thought about using Filters, but they are used for authentications and don't allow to modify response, another way of doing this would be to create custom TypeFormatter, but this involves changing every method.

What I would like to achieve is ability to add attribute to controller or method that would result in encrypted response.

For now I'm using context.Request.Path inside Invoke in my middleware to exclude path I'd like to skip when encrypting, but as I wrote I'd like to use attribute base inclusion/exclusion.

Below is my current middleware code:

public static class ResponseEncrypterMiddlewareExtensions
{
    public static void UseResponseEncrypterMiddleware(this IAppBuilder app)
    {
        app.Use<ResponseEncrypterMiddleware>();
    }
}

public class ResponseEncrypterMiddleware : OwinMiddleware
{
    public ResponseEncrypterMiddleware(OwinMiddleware next) : base(next)
    {
    }

    public override async Task Invoke(IOwinContext context)
    {
        using (var stream = context.Response.Body)
        {
            using (var buffer = new MemoryStream())
            {
                context.Response.Body = buffer;
                await Next.Invoke(context);
                buffer.Seek(0, SeekOrigin.Begin);

                using (var reader = new StreamReader(buffer))
                {
                    string responseBody = await reader.ReadToEndAsync();

                    /*excludes*/
                    if (!string.IsNullOrWhiteSpace(responseBody)
                        && !context.Request.Path.Value.Contains("test1/")
                         && !context.Request.Path.Value.Contains("nothing_special/")
                        )
                    {
                        try
                        {
                            responseBody = /*encrypt here*/;
                        }
                        catch (Exception ex)
                        {
                            Debug.WriteLine(ex);
                            responseBody = string.Empty;
                            context.Response.StatusCode = 500;
                        }
                    }

                    var bytes = Encoding.UTF8.GetBytes(responseBody);
                    buffer.SetLength(0);
                    buffer.Write(bytes, 0, bytes.Length);
                    buffer.Seek(0, SeekOrigin.Begin);
                    context.Response.ContentLength = buffer.Length;
                    buffer.CopyTo(stream);
                }
            }
        }
    }
}

EDIT1: I'm able to modify response using custom ActionFilterAttribute, but this just a partial solution. I know I can include globally that attribute, bit I'd like a way to exclude some actions and controllers (by another attribute), so everything will be encrypt as default, but attribute on controller or action will allow clear (not encrypted) response.

EDIT2: Inside my Startup.cs I'm adding multiple middlewares. First one is responsible for encrypting response and second for tracing, below is my code inside Configuration method:

app.UseResponseEncrypterMiddleware();
app.UseRequestLogger();

Having them added in that order allows me to trace (log) request and response just before encrypting - middlewares start from top to bottom and end in opposite direction.
I've tried creating custom ActionFilter but this encrypts my content before it gets to tracing middleware.