HTTPS only works on localhost

Question

I am trying to setup https on one of my EC2 instances. I've done this many times, but for some reason, for this instance, it refuses to work.

Problem

When I connect to the domain via the local ip, the https works fine. When I connect to the domain via the public ip, the https does not work and the browser repsonse is "ERR CONNECTION REFUSED"

When I reload sometimes, you can see it accepted the https, and then it blocks it immediately, almost as if there is a firewall rule, but there isnt.

normal http works 100% local and public.

I am not blocking any ports at this stage via Security groups on amazon. I have no rules on ip tables and I have disabled ufw.

I have enabled SSL for apache and the test keys works fine locally.

This is the output of sudo netstat -tlnp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      937/sshd
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1060/mysqld
tcp6       0      0 :::22                   :::*                    LISTEN      937/sshd
tcp6       0      0 :::443                  :::*                    LISTEN      2798/apache2
tcp6       0      0 :::80                   :::*                    LISTEN      2798/apache2

This is my /etc/apache2/ports.conf file:

Listen 80
Listen 443

I have enabled the default-ssl site without any errors.

This is my /etc/hosts file:

127.0.0.1 localhost


# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I also tried using 0.0.0.0 instead of 127.0.0.0 and it did nothing.

This is my website's conf file for apache (I replaced my real site name with "mysite.com" for security purposes):

<VirtualHost *:80>
     DocumentRoot /var/www/html/mysite.com
     ServerName mysite.com
     ServerAlias www.mysite.com
        <Directory /var/www/html/mysite.com>
                AllowOverride All
                RewriteEngine On
                Require all granted
                Options -Indexes +FollowSymLinks
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
<VirtualHost *:443>
     DocumentRoot /var/www/html/mysite.com
     ServerName mysite.com
     ServerAlias www.mysite.com
     SSLEngine on
     SSLCertificateFile /usr/local/ssl/public.crt
     SSLCertificateKeyFile /usr/local/ssl/private/private.key
     SSLCACertificateFile /usr/local/ssl/intermediate.crt
</VirtualHost>

This exact same conf file was working on another instance, which is telling me that the problem is not with this file.

What am I missing ? Any help please. Thanks in advance

EDIT

Here are my loaded modules for apache:

sudo apache2ctl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 access_compat_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 filter_module (shared)
 mime_module (shared)
 mpm_prefork_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 socache_shmcb_module (shared)
 ssl_module (shared)
 status_module (shared)
 wsgi_module (shared)

EDIT 2

Some interesting apache error.log entries, not when I restarted the service, just randomly a few minutes earlier (could this be whats causing the problems ?):

[Wed Feb 17 21:04:48.478106 2016] [ssl:warn] [pid 3629] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Wed Feb 17 21:04:48.507277 2016] [ssl:warn] [pid 3630] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Wed Feb 17 21:04:48.507324 2016] [:warn] [pid 3630] mod_wsgi: Compiled for Python/3.4.0.
[Wed Feb 17 21:04:48.507329 2016] [:warn] [pid 3630] mod_wsgi: Runtime using Python/3.4.3.
[Wed Feb 17 21:04:48.509502 2016] [mpm_prefork:notice] [pid 3630] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.14 OpenSSL/1.0.1f mod_wsgi/3.4 Python/3.4.3 configured -- resuming normal operations
[Wed Feb 17 21:04:48.509517 2016] [core:notice] [pid 3630] AH00094: Command line: '/usr/sbin/apache2'

I disabled the wsgi module just to make sure that's not my problem, and restarted apache service, still, no luck, same problem.

EDIT 3

I'm dumping some of my phpinfo() to show more info. SSL is set to YES. I have no idea whats going on :(

EDIT 4

I think that I might need to add the SSL certifiacte on the load balancer for any of this to even work. I'm trying that now, will let everyone know...

Answer 1

The only way to achieve HTTPS traffic to an EC2 instance in a private subnet sitting behind a load balancer within a VPC on AWS, is to have an HTTPS listener on the load balancer (ELB) itself with the certificate of the specific server/domain on the load balancer. In my case, even if you have HTTPS 443 open in your security rules of the load Balancer (inbound, outbound) and on the EC2 instance (inbound, outbound), the ELB still blocks all HTTPS traffic on that port to the EC2 instance if the ELB does not have a listener with a valid certificate. No where in the documentation, as far as I have read, does it explicitly say that. Hope this helps.

Step 1

Submit the certificate from the EC2 instance into IAM using AWS cli.

I added the certificate files on my server and had to upload them to IAM like so (yes, the file:// part is necessary, even on linux):

aws iam upload-server-certificate --server-certificate-name my-server-cert 
--certificate-body file://my-certificate.pem --private-key file://my-private-key.pem 
--certificate-chain file://my-certificate-chain.pem

as explained here : http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ssl-server-cert.html

Step 2

Then you need to setup a listener on the ELB which has the EC2 instance attached to it. The listener is then for HTTPS and port 443. The listener will ask for a certificate and it will have the one you added from the aws cli earlier, already listed. If it is not listed, log out of the AWS console and log back in. After, this, traffic via HTTPS will start flowing to your EC2 instance.

I wasn't sure that this was needed, because I had the ports open for 443 on the load balancer's security group and nowhere in the documentation does it explicitly say that if you use a load balancer, this is the ONLY way of achieving an HTTPS connection. It only explains that this is the way to go when you want a HTTPS load balancer, which can mean two different things, however, I'm glad it's sorted out.

Answer 2

Can you also attach a log to show any Apache error output?

Reorder your virtualhosts and put 443 first.

Its possible you could be affected by this: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/795315