I tried some of the suggestions above- but in my case I had all the versions of EF aligned on version 5.0.9 except for the autogenerated Swashbuckle.
I removed the following:
<PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="6.0.9" />
<PackageReference Include="Microsoft.VisualStudio.Web.CodeGenerators.Mvc" Version="6.0.9" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="6.3.0" />
then I had to reinstall (because it is used by my code):
<PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="6.0.9" />
Finally I ran the View scaffold again and now it worked!
I can see that during scaffolding the following package was auto installed:
<PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="6.0.10" />