Mysterious Http 408 errors in AWS elasticbeanstalk-access_log

Question

The elasticbeanstalk-access_log log-file in our AWS EBS instances are full of 408 errors, like these:

172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:59 +0000] "-" 408 - "-" "-"

They appear randomly, sometimes there are a few minutes between them, sometimes there are 4-6 errors within a few seconds. These errors also happen on our non-public staging environment when there is no any real traffic on the server, so the source of these requests is probably one of AWS's own service.

score 15 · Answer 1 · answered Jun 27 '18 at 16:26

15

I had the similar issue: HTTP error code 408 on AWS Elastic Beanstalk. The solution I had to implement in order to fix this was changing the Instance port and protocol on the Classic LB itself to 80 and HTTP.

Initially both ports and protocols were set to 443 and HTTPS. So make sure you have the instance port and protocol set to 80 even though you have the LB port and protocol set to 443 and HTTPS.

answered Jun 27 '18 at 16:26

Antonio Nedanoski

191
1
5

This worked for me it seems that the default settings needed to be switched for me on the LB; one for 80/HTTP on LB and 80/HTTP on instance, and another 443/HTTPS on the LB, 80/HTTP on the instance. – cwash Aug 20 '20 at 04:23
Dunno how it changed back to 443, never bothered checking till I saw this answer. I have been at it for 2 days now. Thanks – BlackPearl Nov 29 '20 at 13:07

Feltkamp.tv Multimedia · Answer 2 · 2017-11-30T11:42:33.700

EDIT: Do you have a classic load balancer? Change to the application load balancer by creating a new environment with the Elastic Beanstalk cli and select application load balancer. This will solve this issue.

ELB has a mechanism called pre-open connections. The ELB does this so that your requests can be served faster, i.e. your new requests will not have to wait at the ELB for the extra time it will take to open up a fresh connection to the backend before the requests get sent to the backends. If you have a lower keep-alive timeout, causing pre-opened connections to be closed quicker which will make your backend generate a 408 error response to indicate that the connection was closed because the client (ELB) timeout expired without the ELB sending any request on that particular connection.

If you have modified the ELB idle connection timeout, then you need to make sure your http keep-alive timeout value is greater than the ELB idle connection timeout value. If this is not the case enable keep alive timeouts and make sure the value is greater than the ELB idle connection timeout.

You can change the keepalive timeout in apache by adding a .config file in the ebextensions folder with the following code:

files:
  "/etc/httpd/conf.d/keepalive.conf" :
  mode: "000644"
  owner: root
  group: root
  content: |
    # Enable TCP keepalive
    Timeout 300
    KeepAlive On
    MaxKeepAliveRequests 100
    KeepAliveTimeout 300
    <IfModule mod_reqtimeout.c>
     RequestReadTimeout header=300 body=300
    </IfModule>

You have an indentation issue. mode (and below) all need to be indented one more level. — Oded, Sep 13 '19 at 03:00

score 1 · Answer 3 · answered Apr 23 '16 at 23:09

We faced the same issue and the suggestion made at the bottom of this AWS forum thread resolved it. In short, you should make sure that the idle time configured on the Elastic Loadbalancer is slightly lower than the idle timeout configured for the Apache httpd running on each of the instances.

score 0 · Answer 4 · answered Feb 05 '20 at 14:41

It is probably because instance keepalaive timeout is shorter than load balancer. It has to be few seconds longer than load balancer's idle timeout, to do that create config into your .ebextensions folder with this content replacing "YOUR-TIME" with time longer than your instance load balancer's timeout:

files:
  "/etc/httpd/conf.d/mod_reqtimeout.conf" :
    mode: "000644"
    owner: root
    group: root
    content: |
      <IfModule reqtimeout_module>
        RequestReadTimeout header=YOUR-TIME,MinRate=500 body=YOUR-TIME,MinRate=500
      </IfModule>

score -5 · Answer 5 · answered Mar 23 '16 at 09:58

-5

I played with the settings for hours on different environments, and here is the solution:

When I turned off the 'Connection Draining' under Configuration -> Load Balancing, the errors are vanished from the logs.

And here comes the best part: When I turned on the Connection Draining again, the errors didn't come back!

So it seems that turning it off and on works on AWS Load-Balancer too (not only on Windows...)

answered Mar 23 '16 at 09:58

Selindek

3,269
1
18
25

I don't know who and why downvote this answer, but at the time I added this answer AWS did work this way. And I spent a lot of time with this issue at those days, and believe me: Even this solution sounds funny, this was the only working solution at that time for me. And even if it is fixed now it's still could help people to realize that sometimes these simple ideas are still working here or there. – Selindek Nov 18 '18 at 13:42

Mysterious Http 408 errors in AWS elasticbeanstalk-access_log

5 Answers5

Linked