do I need to bother with Server Explorer, or can I just create my databases and use them with SQL Server Object Explorer?
Yes, you could see your database just in SQL Server Object Explorer.
Refer to https://learn.microsoft.com/en-us/aspnet/core/tutorials/first-mvc-app/working-with-sql?view=aspnetcore-2.2&tabs=visual-studio
Also, for some reason my web.config file does not appear in my solution and I was thinking that this is the problem
Asp.net core do not use web.config any more.
When you publish your project, it does create a web.config. If you publish to a folder profile, you will see it there, or if you publish to IIS/Azure you will also see the file using File Explorer or Azure Console.
Refer to web.config in ASP.NET Core 2