I have a .NET core 2.1 web app deployed to a VPS (I have root access) running ubuntu 18.04 on nginx. Everything has been smooth until the last time I updated the files on the server, and I don't understand how it stopped working. It runs flawlessly on my local machine, but somehow on the server the redirection to HTTPS fails.
Program.cs code
namespace MyApp
{
public class Program
{
public static void Main(string[] args)
{
BuildWebHost(args).Run();
}
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.CaptureStartupErrors(true)
.UseSetting("detailedErrors", "true")
.UseStartup<Startup>()
.Build();
}
}
Launchsettings.json
{
"iisSettings": {
"windowsAuthentication": false,
"anonymousAuthentication": true,
"iisExpress": {
"applicationUrl": "http://localhost:55533",
"sslPort": 44388
}
},
"profiles": {
"IIS Express": {
"commandName": "IISExpress",
"launchBrowser": true,
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development"
}
},
"MyApp-Dev": {
"commandName": "Project",
"launchBrowser": true,
"applicationUrl": "http://localhost:5000;https://localhost:5001",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development"
}
}
}
}
Appsettings.json
{
"Kestrel": {
"EndPoints": {
"Http": {
"Url": "http://localhost:5000"
},
"httpsdefaultcert": {
"url": "https://localhost:5001"
}
},
"Logging": {
"IncludeScopes": false,
"LogLevel": {
"Default": "Information"
}
},
"AllowedHosts": "*"
}
Kestrel configuration file
[Unit]
Description=My App
[Service]
WorkingDirectory=/var/www/myapp.com/
ExecStart=/usr/bin/dotnet /var/www/myapp.com/MyApp.dll
Restart=always
RestartSec=10 # Restart service after 10 seconds if dotnet service crashes
KillSignal=SIGINT
SyslogIdentifier=dotnet-myapp.com
User=myuser
Environment=ASPNETCORE_ENVIRONMENT=Production
[Install]
WantedBy=multi-user.target
The first time I had a similar problem, I was able to solve it by replacing the user in this file by "myuser". Previously, because of by following some tutorials, I was using "www-data".
Nginx Configuration files
upstream myapp.com {
server localhost:5000;
}
server {
listen 80;
location ~ /.well-known {
allow all;
root /var/www/well-known;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
# Enable https and http/2
listen *:443 ssl http2;
# The certificate served by Let's encrypt can contain more than one domain which is very convenient
server_name myapp.com;
server_name www.myapp.com;
ssl_certificate /etc/letsencrypt/live/www.myapp.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.myapp.com/privkey.pem; # managed by Certbot
# security
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
# Turn on OCSP stapling as recommended at
# https://community.letsencrypt.org/t/integration-guide/13123
# requires nginx version >= 1.3.7
ssl_stapling on;
ssl_stapling_verify on;
# Uncomment this line only after testing in browsers,
# as it commits you to continuing to serve your site over HTTPS in future
# add_header Strict-Transport-Security "max-age=31536000";
location / {
proxy_pass http://myapp.com;
}
}
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
I did try to change the user in the above file, from "www-data" to "myuser", but since it didn't fix anything, I got back to the previously working configuration.
I am leaving here the steps I have been using to troubleshoot the problem, and similar ones. Any input regarding my approach is more than welcome, even if you don't know how to help me with this specific problem.
1. Firewall : seems to be correctly configured!
sudo ufw status
Status: active
To Action From
-- ------ ----
22/tcp ALLOW Anywhere
80/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
Nginx HTTP ALLOW Anywhere
22/tcp (v6) ALLOW Anywhere (v6)
80/tcp (v6) ALLOW Anywhere (v6)
443/tcp (v6) ALLOW Anywhere (v6)
Nginx HTTP (v6) ALLOW Anywhere (v6)
2. Kestrel : Seems to be up and running! Buuut...
sudo systemctl status kestrel-myApp.service
● kestrel-myApp.service - My App
Loaded: loaded (/etc/systemd/system/kestrel-myApp.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2019-02-16 06:46:25 UTC; 1h 6min ago
Main PID: 1951 (dotnet)
Tasks: 19 (limit: 4636)
CGroup: /system.slice/kestrel-myApp.service
└─1951 /usr/bin/dotnet /var/www/myApp/MyApp.dll
[ERROR HINT] Nonetheless, when I did check the logs, I was able to find this:
Feb 16 06:46:26 localhost dotnet-myapp.com[1951]: User profile is available. Using '/home/myuser/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
Feb 16 06:46:26 localhost dotnet-myapp.com[1951]: info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0]
Feb 16 06:46:25 localhost systemd[1]: Started My App.
Feb 16 06:46:20 localhost systemd[1]: /etc/systemd/system/kestrel-myapp.com.service:9: Failed to parse sec value, ignoring: 10 # Restart service after 10 seconds if dotnet service crashes
Feb 16 06:45:28 localhost systemd[1]: Failed to start My App.
Feb 16 06:45:28 localhost systemd[1]: kestrel-myapp.com.service: Failed with result 'signal'.
Feb 16 06:45:28 localhost systemd[1]: kestrel-myapp.com.service: Start request repeated too quickly.
Feb 16 06:45:28 localhost systemd[1]: Stopped My App.
Feb 16 06:45:28 localhost systemd[1]: kestrel-myapp.com.service: Scheduled restart job, restart counter is at 5.
Feb 16 06:45:28 localhost systemd[1]: kestrel-myapp.com.service: Service hold-off time over, scheduling restart.
Feb 16 06:45:28 localhost systemd[1]: kestrel-myapp.com.service: Failed with result 'signal'.
Feb 16 06:45:28 localhost systemd[1]: kestrel-myapp.com.service: Main process exited, code=killed, status=6/ABRT
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at MyApp.Program.Main(String[] args) in C:\Users\myuser\Documents\Visual Studio 2017\Projects\MyApp\MyApp\Program.cs:line 34
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host)
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token)
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String shutdownMessage)
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at Microsoft.AspNetCore.Hosting.Internal.WebHost.StartAsync(CancellationToken cancellationToken)
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.ValidateOptions()
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Load()
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: For more information on configuring HTTPS see https://go.microsoft.com/fwlink/?linkid=848054.
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: To generate a developer certificate run 'dotnet dev-certs https'. To trust the certificate (Windows and macOS only) run 'dotnet dev-certs https --trust'.
Feb 16 06:45:28 localhost dotnet-myapp.com[1893]: Unhandled Exception: System.InvalidOperationException: Unable to configure HTTPS endpoint. No server certificate was specified, and the default developer certificate could not be found.
I was able to see this problem a few times in the logs, but it didn't happen in the last few hours. And the website remains inaccessible! I think this is the root of problem though, and I don't know how to fix it
3. Nginx : Seems to be up and running, buuut...
sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2019-02-16 06:32:05 UTC; 1h 37min ago
Docs: man:nginx(8)
Main PID: 745 (nginx)
Tasks: 4 (limit: 4636)
CGroup: /system.slice/nginx.service
├─ 745 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─ 746 nginx: worker process is shutting down
├─5455 nginx: worker process
└─5456 nginx: worker process
sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[ERROR HINT] When I go in the logs, I do see some errors, which don't happen every time I fail to access the website:
2019/02/16 09:31:04 [error] 793#793: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 123.2.45.130, server: myapp.com, request: "GET / HTTP/2.0", upstream: "http://127.0.1.1:5000/", host: "myapp.com"
4. Certificates : Seem alright!
openssl s_client -connect localhost:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = www.myapp.com
verify return:1
---
Certificate chain
0 s:/CN=www.myapp.com
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
(...)
-----END CERTIFICATE-----
subject=/CN=www.myapp.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 3634 bytes and written 334 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session: (...)
Also, I get A when I try this out: https://whatsmychaincert.com
5. Try to retrieve content from localhost :
[ERROR HINT]
--2019-02-16 08:26:10-- http://localhost:5000/
Resolving localhost (localhost)... 127.0.0.1, 127.0.1.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:5000... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://localhost:5001/ [following]
--2019-02-16 08:26:10-- https://localhost:5001/
Connecting to localhost (localhost)|127.0.0.1|:5001... connected.
ERROR: cannot verify localhost's certificate, issued by ‘CN=localhost’:
Unable to locally verify the issuer's authority.
To connect to localhost insecurely, use `--no-check-certificate'.
Running the command insecurely, as suggested in the output, works fine!
wget http://localhost:5000 --no-check-certificate
SOURCES
In order to have the server configured and the app deployed, I used the following sources
https://www.meziantou.net/2017/04/25/publishing-an-asp-net-core-website-to-a-linux-host
https://odan.github.io/2018/07/17/aspnet-core-2-ubuntu-setup.html
https://learn.microsoft.com/en-gb/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-2.2
