I would not like to display the ValidationSummary in case it only displays already displayed field related errors. However I do need ValidationSummary when custom server side validation error occurs like:
if (!UserManager.IsEmailConfirmed(user.Id))
{
AuthenticationManager.SignOut();
ModelState.AddModelError("", "You need to confirm your email.");
return View(model);
}