How to encrypt traffic between the load balancer and web servers with elastic beanstalk

Question

I want to encrypt traffic between my load balancer and web servers in an Elastic Beanstalk environment. Amazon has a guide here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/configuring-https-endtoend.html but it involves manually generating a certificate for your servers. Is there a fully automatic alternative?

Answer 1

If you have your servers generate their own self-signed certificate as part of the deployment container commands, then each server will get an updated certificate every time you deploy and when a new server is started.

The best command I have found for this is the following, which creates certificates valid for 10 years:

sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/pki/tls/certs/server.key -out /etc/pki/tls/certs/server.crt -days 3650 -nodes -subj "/CN=example.com"

Using this approach, as long as you deploy (including upgrading your EB container version) at least once a decade, your servers will stay up.

This drastically simplifies the setup for this as well. Now all you need to do is the following:

1. Add a config file to your elastic beanstalk project which generates self-signed certificates and adds HTTPS settings to the web server.
2. Have the web server security group accept port 443 connections from the load balancer security group.
3. Set your load balancer to forward traffic from port 443 to port 443.

Below is an example of a full HTTPS elastic beanstalk config file for python. This is a slight modification of AWS's suggested config file for python. I've added the generate certificate command to the beginning of container commands and removed the two file statements for /etc/pki/tls/certs/server.crt and /etc/pki/tls/certs/server.key as they are now auto generated. AWS examples for other languages can be found here.

AWS Linux 2, Apache-based deployment

Put the following in .ebextensions/ssl.config:

container_commands:
  01_create_certs:
    command: |
      sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/pki/tls/certs/server.key -out /etc/pki/tls/certs/server.crt -days 3650 -nodes -subj "/CN=example.com"
  02_restart_httpd:
    command: |
      # Condition on whether httpd is running for compatibility with EB worker environments
      sudo systemctl status httpd && sudo systemctl restart httpd || echo "httpd not running"
  03_wait_for_httpd_restart:
    command: "sleep 3"

Put the following in .platform/httpd/conf.d/ssl.conf:

Listen 443
<VirtualHost *:443>
  SSLEngine on
  SSLCertificateFile "/etc/pki/tls/certs/server.crt"
  SSLCertificateKeyFile "/etc/pki/tls/certs/server.key"
  # Limit requests to 100 MB
  LimitRequestBody 104857600

  <Proxy *>
    Require all granted
  </Proxy>
  ProxyPass / http://localhost:8000/ retry=0
  ProxyPassReverse / http://localhost:8000/
  ProxyPreserveHost on
</VirtualHost>

AWS Linux 1, Apache-based deployment

Put the following in .ebextensions/ssl.config:

packages:
  yum:
    mod24_ssl : []
    
files:
  /etc/httpd/conf.d/ssl.conf:
    mode: "000644"
    owner: root
    group: root
    content: |
      LoadModule wsgi_module modules/mod_wsgi.so
      WSGIPythonHome /opt/python/run/baselinenv
      WSGISocketPrefix run/wsgi
      WSGIRestrictEmbedded On
      Listen 443
      <VirtualHost *:443>
        SSLEngine on
        SSLCertificateFile "/etc/pki/tls/certs/server.crt"
        SSLCertificateKeyFile "/etc/pki/tls/certs/server.key"
        
        Alias /static/ /opt/python/current/app/static/
        <Directory /opt/python/current/app/static>
        Order allow,deny
        Allow from all
        </Directory>
        
        WSGIScriptAlias / /opt/python/current/app/application.py
        
        <Directory /opt/python/current/app>
        Require all granted
        </Directory>
        
        WSGIDaemonProcess wsgi-ssl processes=1 threads=15 display-name=%{GROUP} \
          python-path=/opt/python/current/app \
          python-home=/opt/python/run/venv \
          home=/opt/python/current/app \
          user=wsgi \
          group=wsgi
        WSGIProcessGroup wsgi-ssl
        
      </VirtualHost>
      
container_commands:
  01_create_certs:
    command: |
      sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/pki/tls/certs/server.key -out /etc/pki/tls/certs/server.crt -days 3650 -nodes -subj "/CN=example.com"
  02_kill_httpd:
    command: "sudo restart supervisord"
  03_wait_for_httpd_death:
    command: "sleep 3"

Answer 2

If you are doing this for a Classic Load Balancer in Apache-based AWS Linux 2 as opposed to a shinier, newer Application Load Balancer, in addition to adding the config files listed here, you will also want to perform the following steps:

1. Put the following in .platform/httpd/conf.d/keepalive.conf:

# Enable TCP keepalive
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 300
<IfModule mod_reqtimeout.c>
 RequestReadTimeout header=300 body=300
</IfModule>

2. Ensure that your ELB is configured to route HTTP traffic to port 80 via HTTP and HTTPS traffic to port 443 via HTTPS.

This will prevent ELB pre-connections from being wrongfully terminated, which if left unchecked will pollute your health-check data (for more on this, see Mysterious Http 408 errors in AWS elasticbeanstalk-access_log). In my testing, not performing this step resulted in a 25% false negative rate for automated health checks.

You may also wish to consider adding an SSL rewrite file to the config to reroute all incoming insecure traffic to the secure port, e.g. .platform/httpd/conf.d/ssl_rewrite.conf with the contents:

RewriteEngine On
<If "-n '%{HTTP:X-Forwarded-Proto}' && %{HTTP:X-Forwarded-Proto} != 'https'">
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R,L]
</If>