Why am I getting an Apache Proxy 503 error?

Question

My server was doing just fine up until yesterday. It was running Redmine, and it was the happiest little server until my "friend" imported a SQL table that my little guy couldn't take. Unfortunately, after an hour of trying to get the lil guy to respond, we had to power cycle him.

Now after restart, we get a 503 error when trying to visit the domain connected to Redmine. It's hooked up to a Mongrel daemon, and we use Apache Proxy to direct all connections to the port Redmine is running on.

Using Lynx on the server (http://localhost:8000) you can see the Ruby application working fine. But this bit is not working in my Apache configuration file:

<VirtualHost *:80>
    ServerName sub.example.com
    ProxyPass / http://localhost:8000
    ProxyPassReverse / http://localhost:8000
    ProxyPreserveHost on
    LogLevel debug
</VirtualHost>

Here's the error log output for Apache:

[debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //localhost:8000
[debug] proxy_util.c(1335): [client 216.27.137.51] proxy: http: found worker http://localhost:8000 for http://localhost:8000/
[debug] mod_proxy.c(756): Running scheme http handler (attempt 0)
[debug] mod_proxy_http.c(1687): proxy: HTTP: serving URL http://localhost:8000/
[debug] proxy_util.c(1755): proxy: HTTP: has acquired connection for (localhost)
[debug] proxy_util.c(1815): proxy: connecting http://localhost:8000/ to localhost:8000
[debug] proxy_util.c(1908): proxy: connected / to localhost:8000
[debug] proxy_util.c(2002): proxy: HTTP: fam 2 socket created to connect to localhost

[error] (13)Permission denied: proxy: HTTP: attempt to connect to 127.0.0.1:8000 (localhost) failed
[error] ap_proxy_connect_backend disabling worker for (localhost)

[debug] proxy_util.c(1773): proxy: HTTP: has released connection for (localhost)

Hey Please revert on below post. As its similar error http://stackoverflow.com/questions/9461086/tomcat-application-not-responding-with-no-logs/23710275#23710275 — Ranjeet Ranjan, May 17 '14 at 10:38

score 97 · Accepted Answer · answered Feb 08 '12 at 14:56

Apache will respond with 503's for at least 60 seconds any time it detects that the backend server is down. This is the default behavior. As in your example, if you restart your backend server (Rails in this example) and someone tries to access it through the Apache proxy before Rails is ready then Apache will return 503's for the next 60 seconds regardless if your backend is now 'up'. Please see the apache docs on ProxyPass where it states:

retry 60

Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance, and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.

So if you set your Proxy Pass to include retry=0 you won't see the 503's when you restart your backend service. This is also useful when using Apache as reverse proxy during development! For example:

ProxyPass / http://localhost:8000 retry=0

For reference, a link to [ProxyPass docs for Apache 2.2](http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass). — Pistos, Jul 23 '12 at 16:01

score 54 · Answer 2 · edited Nov 18 '11 at 16:51

54

Run following command

# /usr/sbin/setsebool httpd_can_network_connect 1

OR

# /usr/sbin/setsebool httpd_can_network_connect true

and after that restart httpd

# service httpd restart

edited Nov 18 '11 at 16:51

animuson

53,861
28
137
147

answered Nov 18 '11 at 16:03

user626710

556
5
4

8

setsebool -P ... makes the changes persist. – Steve Bennett Mar 15 '12 at 01:23
3

This doesn't help to me. Apache + mongrel This is disabling the SELinux and I can't see any relation with the question. – Ivailo Bardarov Aug 05 '13 at 19:52
1

this help me but I don't understand anything, could you please explain? – Wasim A. Oct 12 '20 at 17:00
1

`/usr/sbin/setsebool: No such file or directory` for me. I can see that there is no file named `setsebool` in that directory. – GreatBlakes Aug 02 '21 at 02:56

Andrew Flanagan · Answer 3 · 2009-03-25T19:30:56.703

Are you sure they're restarting in the correct order? I've had weird issues where Apache starts, then Mongrel starts and although Mongrel is running, Apache still throws the proxy error.

I've solved this in the past with various incantations and restarts of Apache and eventually the gods are happy. It seems that sometimes the Mongrel processes don't properly shut down so you have to manually kill them. Here's a link with some [possible] help.

I ended up adding a "kill" option to my /etc/init.d/ mongrel script because it happened so much. It stop Mongrel, killed all Mongrel sessions, started Mongrel and restarted Apache.

<snip>
    kill)
      echo "Stopping, killing, starting, and restarting Apache..."
      mongrel_cluster_ctl stop -c $CONF_DIR --clean
      killall -u mongrel
      mongrel_cluster_ctl start -c $CONF_DIR --clean
      /etc/init.d/httpd restart
      RETVAL=$?
  ;;
</snip>

Probably not a very good solution but the evil went away.

exshovelrydr's answer explains why start order matters, and how to change the Apache configuration so that start order *doesn't* matter. — Warren Young, May 14 '14 at 19:41

score 1 · Answer 4 · answered Mar 26 '09 at 01:06

Try running monit to monitor your mongrels behind Apache, and that way it can restart mongrels for you if they die or get too hungry for memory. If for any reason Apache still gets confused you may just have to gracefully restart apache and it should resolve itself, but for 99% of cases having monit watch over your mongrels should avoid this happening again. The other option is look into Phusion Passenger.

score 0 · Answer 5 · answered Oct 21 '15 at 08:06

0

First check whether the port 8080 is listening or not by the following command

netstat -tlpn

If not than restart the jenkins server by the following command

sudo /etc/init.d/jenkins start

It should work now. Hope it helps.

answered Oct 21 '15 at 08:06

Ajeet Khan

8,582
8
42
65

D. Hung · Answer 6 · 2017-07-14T08:58:15.487

0

Fist, you must install selinux: (SELinux stands for Security-Enhanced Linux.)

apt-get install selinux

After that, you can enable Security Policy of SElinux by follow command:

sed -i 's/SELINUX=.*/SELINUX=permissive/' /etc/selinux/config

Notice:

#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.

Final,restart apache!

edited Jul 14 '17 at 08:58

answered Jun 14 '17 at 08:45

D. Hung

1
2

3

Your answer changes the SELINUX setting twice? Turning off selinux is already the accepted answer - I'm not sure what you're adding here. Also, please comment/explain what your answer does - this improves the long term value of SO. – Andy Jones Jun 14 '17 at 20:03
Sorry, English is not my native language, so I'm slightly stuck in text interpretation :) – D. Hung Jul 14 '17 at 08:47
1

- I have encountered this error and I fixed it with the above statement. SELinux stands for Security-Enhanced Linux. It is a way to improve the server security. When SELinux is enable, Apache will using SELinux policy instead of Apache policy. – D. Hung Jul 14 '17 at 08:54

Why am I getting an Apache Proxy 503 error?

6 Answers6

Linked