Custom errors on specific MVC 5 controllers

Question

I am having an issue where I want to display custom 404 and 500 views only for specific controllers.

The reason I want to do this is that we have APIControllers and standard Controllers in the same project and I do not want to rewrite the API error responses.

I am trying to achieve this with a custom attribute which inherits HandleErrorAttribute, but I cannot seem to get 404's to go through it. Here is what I have for OnException so far:

public override void OnException(ExceptionContext filterContext)
    {
        if (filterContext.ExceptionHandled || !filterContext.HttpContext.IsCustomErrorEnabled)
            return;

        //Defaults to 500 if it cannot be determined
        int statusCode = new HttpException(null, filterContext.Exception).GetHttpCode();

        //We only want to capture 500 and 404 at this stage
        switch (statusCode)
        {
            case 404:
                View = _404ViewName;
                break;
            case 500:
                View = _500ViewName;
                break;
            default:
                return;
        }

        Master = _layoutViewName;

        string controllerName = (string)filterContext.RouteData.Values["controller"];
        string actionName = (string)filterContext.RouteData.Values["action"];

        HandleErrorInfo model = new HandleErrorInfo(filterContext.Exception, controllerName, actionName);
        ViewDataDictionary viewData = new ViewDataDictionary<HandleErrorInfo>(model);

        filterContext.Result = new ViewResult
        {
            ViewName = View,
            MasterName = Master,
            ViewData = viewData,
            TempData = filterContext.Controller.TempData
        };

        filterContext.ExceptionHandled = true;
        filterContext.HttpContext.Response.Clear();
        filterContext.HttpContext.Response.StatusCode = statusCode;

        // Certain versions of IIS will sometimes use their own error page when
        // they detect a server error. Setting this property indicates that we
        // want it to try to render ASP.NET MVC's error page instead.
        filterContext.HttpContext.Response.TrySkipIisCustomErrors = true;
    }

I have previously tried this in the config file:

<httpErrors errorMode="DetailedLocalOnly" existingResponse="Replace">
  <remove statusCode="404"/>
  <error statusCode="404" responseMode="ExecuteURL" path="/Error/404"/>
  <remove statusCode="500"/>
  <error statusCode="500" responseMode="ExecuteURL" path="/Error/500"/>
</httpErrors>

But that rewrote the API error responses.

I currently only use the <customErrors mode="On" /> tag instead which works great for 500 errors but I get generic IIS errors for 404s using that.

If I add <httpErrors errorMode="Custom" existingResponse="PassThrough" /> into the web config, the 500 still gives me my custom message view, but the 404 now just shows a blank page.

What do I need to do in order to get 404s to go through my custom attribute as well?

Or if this is not possible, what is a different approach I can take that is not going to affect the API Controllers?