So I had a very long post related to deployment of an elixir app to Elastic Beanstalk (here: HTTP 500 Deploying Elixir/Phoenix to AWS Elastic Beanstalk).
I've finally boiled the problem to my NGINX being misconfigured to route traffic. It's posting this:
2017/11/18 00:52:45 [error] 2811#0: *15 connect() failed (113: No route to host)
while connecting to upstream, client: 172.31.25.36, server: ,
request: "GET / HTTP/1.1",
upstream: "http://172.17.0.2:4000/", host: "172.31.17.239"
I've looked online for possible solutions and found (I also need to add the ability to use websockets that this bit of code looks closest: https://scopestar.com/blog/aws/elasticbeanstalk/websockets/2016/10/21/enable-websockets-on-elasticbeanstalk-nginx-proxy.html
I've gotten the following so far - it deletes the previous config, writes a new one with websockets.
files:
"/etc/nginx/conf.d/01_nginx_websocket.conf":
mode: "000644"
owner: root
group: root
content: |
upstream nodejs {
server 127.0.0.1:8081;
keepalive 256;
}
server {
listen 8080;
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}
access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
location / {
proxy_pass http://nodejs;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
gzip on;
gzip_comp_level 4;
gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
}
container_commands:
removeconfig:
command: "rm -f /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf"
But I'm a little stuck on how I do the actual routing. Is there a good example of how to change the upstream and server routes so that they handshake correctly and solve the error I'm getting?
EDIT:
This website: https://dennisreimann.de/articles/phoenix-nginx-config.html
Suggested I do something along the lines of take my image on my Dockerfile and route all traffic to localhost:myport and then route that to AWSEB:80, which seemed straightforward enough.
I made something that looks like the following in my .ebextensions as .conf:
files:
"/etc/nginx/conf.d/01_nginx_websocket.conf":
mode: "000644"
owner: root
group: root
content: |
upstream nodejs {
server localhost:4000;
keepalive 256;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name newslyproduction3.us-west-2.elasticbeanstalk.com;
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}
access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
location / {
proxy_pass http://nodejs;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
gzip on;
gzip_comp_level 4;
gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
}
container_commands:
removeconfig:
command: "rm -f /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf"
However, this gives different errors and I'm not sure if I'm getting closer or moving farther away.
INFO: Successfully built aws_beanstalk/staging-app
ERROR: Failed to start nginx, abort deployment
ERROR: [Instance: i-0c48e103c226ca0a9] Command failed on instance. Return code: 1 Output: (TRUNCATED)...enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
nginx: configuration file /etc/nginx/nginx.conf test failed
Failed to start nginx, abort deployment.
This post Nginx getting error with specific domain name suggests that I need to increase the name size in the http block, but I don't know where that is in the nginx of AWS EB.
EDIT:
Following code snippets that I found from here https://dennisreimann.de/articles/phoenix-nginx-config.html (and elsewhere I remade my file to look like the following):
files:
"/etc/nginx/conf.d/01_nginx_websocket.conf":
mode: "000644"
owner: root
group: root
content: |
upstream nodejs {
server localhost:4000;
keepalive 256;
}
# log_format healthd '$msec"$uri"'
# '$status"$request_time"$upstream_response_time"'
# '$http_x_forwarded_for';
server {
listen 80;
server_name _ localhost;
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}
access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
access_log /var/log/nginx/access.log;
location / {
proxy_pass http://newslyproduction3.us-west-2.elasticbeanstalk.com;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
gzip on;
gzip_comp_level 4;
gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
}
container_commands:
removeconfig:
command: "rm -f /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf"
So now it is referencing localhost on the docker machine and routing to my instance. Or so I thought.
This uploads ok, but then fails with the following logs for NGINX:
-------------------------------------
/var/log/nginx/access.log
-------------------------------------
172.31.35.70 - - [18/Nov/2017:04:12:30 +0000] "GET / HTTP/1.1" 463 0 "-" "ELB-HealthChecker/2.0"
172.31.6.206 - - [18/Nov/2017:04:12:30 +0000] "GET / HTTP/1.1" 463 0 "-" "ELB-HealthChecker/2.0"
172.31.35.70 - - [18/Nov/2017:04:12:30 +0000] "GET / HTTP/1.1" 463 0 "-" "ELB-HealthChecker/2.0"
172.31.6.206 - - [18/Nov/2017:04:12:30 +0000] "GET / HTTP/1.1" 463 0 "-" "ELB-HealthChecker/2.0"
and my app starts powercycling
-------------------------------------
/var/log/eb-docker/containers/eb-current-app/20c6293b969d-stdouterr.log
-------------------------------------
** (exit) exited in: :gen_server.call(#PID<0.221.0>, {:checkout, #Reference<0.0.1.3329>, true, :infinity}, 5000)
** (EXIT) time out
(db_connection) lib/db_connection/poolboy.ex:112: DBConnection.Poolboy.checkout/3
(db_connection) lib/db_connection.ex:920: DBConnection.checkout/2
(db_connection) lib/db_connection.ex:742: DBConnection.run/3
(db_connection) lib/db_connection.ex:1133: DBConnection.run_meter/3
(db_connection) lib/db_connection.ex:584: DBConnection.prepare_execute/4
** (exit) exited in: :gen_server.call(#PID<0.221.0>, {:checkout, #Reference<0.0.1.3329>, true, :infinity}, 5000)
** (EXIT) time out
(db_connection) lib/db_connection/poolboy.ex:112: DBConnection.Poolboy.checkout/3
(db_connection) lib/db_connection.ex:920: DBConnection.checkout/2
(db_connection) lib/db_connection.ex:742: DBConnection.run/3
(db_connection) lib/db_connection.ex:1133: DBConnection.run_meter/3
(db_connection) lib/db_connection.ex:584: DBConnection.prepare_execute/4
The dockerfile compiles and it runs locally (with non-production configs) and I have the production configs and dockerfile that the tutorial I am working with says to have (https://robots.thoughtbot.com/deploying-elixir-to-aws-elastic-beanstalk-with-docker), so the only thing that I can think is that this has to be some sort of NGINX error.
I feel I'm a bit closer, and trying to route the localhost to the AWS EB endpoint is the right direction.
EDIT:
I've made a question on ServerFault as well, as they may be a more targeted community and have some ideas https://serverfault.com/questions/884036/how-do-i-modify-nginx-routing-on-elastic-beanstalk-aws-so-x-post.
