503 service unavailable. Apache crashes once in 7-10 days

Question

I use Apache server on Ubuntu 14.04. In order redirect all requests from mypage.com to mypage.com:8080/myapp I use mod_proxy. My 000-default.conf looks as follows:

<VirtualHost *:80>
    # The ServerName directive sets the request scheme, hostname and port that
    # the server uses to identify itself. This is used when creating
    # redirection URLs. In the context of virtual hosts, the ServerName
    # specifies what hostname must appear in the request's Host: header to
    # match this virtual host. For the default virtual host (this file) this
    # value is not decisive as it is used as a last resort host regardless.
    # However, you must set it for any further virtual host explicitly.
    #ServerName www.example.com

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html

    # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
    # error, crit, alert, emerg.
    # It is also possible to configure the loglevel for particular
    # modules, e.g.
    #LogLevel info ssl:warn

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
    ProxyPass / http://mypage.com:8080/myapp/ retry=0 timeout=10
    ProxyPassReverse / http://mypage.com:8080/myapp/

    ServerName mypage.com

    # For most configuration files from conf-available/, which are
    # enabled or disabled at a global level, it is possible to
    # include a line for only one particular virtual host. For example the
    # following line enables the CGI configuration for this host only
    # after it has been globally disabled with "a2disconf".
    #Include conf-available/serve-cgi-bin.conf
 </VirtualHost>

And that is my ports.conf

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen 80
Listen 81

<IfModule ssl_module>
    Listen 443
</IfModule>

<IfModule mod_gnutls.c>
    Listen 443
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

And everything works fine until in a several days I always get 503 Service Unavailable error page. and this is written in error.log before error occurs

[Sun Oct 25 21:22:27.779422 2015] [core:error] [pid 14595] [client 38.104.233.252:45239] AH00126: Invalid URI in request GET HTTP/1.1 HTTP/1.1
[Mon Oct 26 02:30:20.925532 2015] [core:error] [pid 15416] [client 190.252.95.102:53508] AH00126: Invalid URI in request GET: index.php HTTP/1.0
[Mon Oct 26 08:22:27.771751 2015] [proxy:error] [pid 15402] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.1.1:8080 (mypage.com) failed
[Mon Oct 26 08:22:27.774040 2015] [proxy:error] [pid 15402] AH00959: ap_proxy_connect_backend disabling worker for (mypage.com) for 0s
[Mon Oct 26 08:22:27.774111 2015] [proxy_http:error] [pid 15402] [client 5.28.160.27:48692] AH01114: HTTP: failed to make connection to backend: mypage.com
[Mon Oct 26 10:55:51.655213 2015] [proxy:error] [pid 15406] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.1.1:8080 (mypage.com) failed
[Mon Oct 26 10:55:51.655422 2015] [proxy:error] [pid 15406] AH00959: ap_proxy_connect_backend disabling worker for (mypage.com) for 0s
[Mon Oct 26 10:55:51.655470 2015] [proxy_http:error] [pid 15406] [client 169.57.5.20:60000] AH01114: HTTP: failed to make connection to backend: mypage.com
[Mon Oct 26 13:32:00.442031 2015] [proxy:error] [pid 15408] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.1.1:8080 (mypage.com) failed
[Mon Oct 26 13:32:00.442356 2015] [proxy:error] [pid 15408] AH00959: ap_proxy_connect_backend disabling worker for (mypage.com) for 0s
[Mon Oct 26 13:32:00.442413 2015] [proxy_http:error] [pid 15408] [client 217.168.189.15:50222] AH01114: HTTP: failed to make connection to backend: mypage.com, referer: http://mypage.com/admn
[Mon Oct 26 13:32:12.245050 2015] [proxy:error] [pid 15402] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.1.1:8080 (mypage.com) failed
[Mon Oct 26 13:32:12.245179 2015] [proxy:error] [pid 15402] AH00959: ap_proxy_connect_backend disabling worker for (mypage.com) for 0s
[Mon Oct 26 13:32:12.245190 2015] [proxy_http:error] [pid 15402] [client 217.168.189.15:50225] AH01114: HTTP: failed to make connection to backend: mypage.com, referer: http://mypage.com/admn
[Mon Oct 26 13:34:44.343614 2015] [mpm_prefork:notice] [pid 991] AH00169: caught SIGTERM, shutting down
[Mon Oct 26 13:36:07.703185 2015] [mpm_prefork:notice] [pid 991] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.11 configured -- resuming normal operations
[Mon Oct 26 13:36:07.708761 2015] [core:notice] [pid 991] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 27 01:20:23.236327 2015] [mpm_prefork:notice] [pid 991] AH00169: caught SIGTERM, shutting down
[Tue Oct 27 01:20:24.265160 2015] [mpm_prefork:notice] [pid 3247] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.11 configured -- resuming normal operations
[Tue Oct 27 01:20:24.265243 2015] [core:notice] [pid 3247] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 27 01:41:17.177723 2015] [mpm_prefork:notice] [pid 3247] AH00169: caught SIGTERM, shutting down
[Tue Oct 27 01:41:18.219244 2015] [mpm_prefork:notice] [pid 11636] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.11 configured -- resuming normal operations
[Tue Oct 27 01:41:18.219351 2015] [core:notice] [pid 11636] AH00094: Command line: '/usr/sbin/apache2'

After I restart my machine it works fine again. Then in 7-10 days it crashes again. Tried a lots of things in internet but always get this error at most in several weeks. Struggling with already for some months. SELinux is disabled.

score 2 · Answer 1 · answered Oct 27 '15 at 07:22

The problem is not with Apache because its still serving a 503 page

The problem is with the application running on port number 8080, you are reverse proxying / to http://mypage.com:8080/myapp/ and when Apache is trying to proxy the request its not getting back the response from the 8080 app and so its trowing the message 'Service Unavailable' . Its inferred from the underlying part of the log.

[Mon Oct 26 13:32:00.442031 2015] [proxy:error] [pid 15408] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.1.1:8080 (mypage.com) failed
[Mon Oct 26 13:32:00.442356 2015] [proxy:error] [pid 15408] AH00959: ap_proxy_connect_backend disabling worker for (mypage.com) for 0s
[Mon Oct 26 13:32:00.442413 2015] [proxy_http:error] [pid 15408] [client 217.168.189.15:50222] AH01114: HTTP: failed to make connection to backend: mypage.com, referer: http://mypage.com/admn
[Mon Oct 26 13:32:12.245050 2015] [proxy:error] [pid 15402] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.1.1:8080 (mypage.com) failed

So check and post(if necessary) the logs of the server running on port number 8080

Thanks for your response.On 8080 works tomcat server deploying java application.Which log file do I have to check?There is something suspicious in catalina.out `Oct 25, 2015 7:49:23 AM org.apache.catalina.realm.LockOutRealm authenticate WARNING: An attempt was made to authenticate the locked user "admin" Oct 25, 2015 7:49:24 AM org.apache.catalina.realm.LockOutRealm authenticate WARNING: An attempt was made to authenticate the locked user "admin"` Do not know if it has any relation to my problem. Also what do u think, about the fact that mostly it works fine,until it crashes after c.a a week — FarizHuseynli, Oct 27 '15 at 22:48
looks like someone is trying to bruteforce into the application, check this thread http://stackoverflow.com/questions/15337296/how-to-fix-tomcat-warning-an-attempt-was-made-to-authenticate-the-locked-user — Shiva, Oct 28 '15 at 07:54

503 service unavailable. Apache crashes once in 7-10 days

1 Answers1