(13: Permission denied) while connecting to upstream:[nginx]

Question

I am working with configuring Django project with Nginx and Gunicorn.

While I am accessing my port gunicorn mysite.wsgi:application --bind=127.0.0.1:8001 in Nginx server, I am getting the following error in my error log file;

2014/05/30 11:59:42 [crit] 4075#0: *6 connect() to 127.0.0.1:8001 failed (13: Permission denied) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "localhost:8080"

Below is the content of my nginx.conf file;

server {
    listen 8080;
    server_name localhost;
    access_log  /var/log/nginx/example.log;
    error_log /var/log/nginx/example.error.log;

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
    }
}

In the HTML page I am getting 502 Bad Gateway.

What mistake am I doing?

Answer 1

Disclaimer

Make sure there are no security implications for your use-case before running this.

Answer

I had a similar issue getting Fedora 20, Nginx, Node.js, and Ghost (blog) to work. It turns out my issue was due to SELinux.

This should solve the problem:

setsebool -P httpd_can_network_connect 1

Details

I checked for errors in the SELinux logs:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied

And found that running the following commands fixed my issue:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx
sudo semodule -i mynginx.pp

Option #2 (probably more secure)

setsebool -P httpd_can_network_relay 1

https://security.stackexchange.com/questions/152358/difference-between-selinux-booleans-httpd-can-network-relay-and-httpd-can-net

References

http://blog.frag-gustav.de/2013/07/21/nginx-selinux-me-mad/
https://wiki.gentoo.org/wiki/SELinux/Tutorials/Where_to_find_SELinux_permission_denial_details
http://wiki.gentoo.org/wiki/SELinux/Tutorials/Managing_network_port_labels

Answer 2

I’ve run into this problem too. Another solution is to toggle the SELinux boolean value for httpd network connect to on (Nginx uses the httpd label).

setsebool httpd_can_network_connect on

To make the change persist use the -P flag.

setsebool httpd_can_network_connect on -P

You can see a list of all available SELinux booleans for httpd using

getsebool -a | grep httpd

Answer 3

I have solved my problem by running my Nginx as the user I'm currently logged in with, mulagala.

By default the user as nginx is defined at the very top section of the nginx.conf file as seen below;

user nginx; # Default Nginx user

Change nginx to the name of your current user - here, mulagala.

user mulagala; # Custom Nginx user (as username of the current logged in user)

However, this may not address the actual problem and may actually have casual side effect(s).

For an effective solution, please refer to Joseph Barbere's solution.

Answer 4

Had a similar problem on Centos 7. When I tried to apply the solution prescribed by Sorin, I started moving in cycles. First I had a permission {write} denied. Then when I solved that I had a permission { connectto } denied. Then back again to permission {write } denied.

Following @Sid answer above of checking the flags using getsebool -a | grep httpd and toggling them I found that in addition to the httpd_can_network_connect being off. http_anon_write was also off resulting in permission denied write and permission denied {connectto}

type=AVC msg=audit(1501830505.174:799183): avc:  
denied  { write } for  pid=12144 comm="nginx" name="myroject.sock" 
dev="dm-2" ino=134718735 scontext=system_u:system_r:httpd_t:s0 
tcontext=system_u:object_r:default_t:s0 tclass=sock_file

Obtained using sudo cat /var/log/audit/audit.log | grep nginx | grep denied as explained above.

So I solved them one at a time, toggling the flags on one at a time.

setsebool httpd_can_network_connect on -P

Then running the commands specified by @sorin and @Joseph above

sudo cat /var/log/audit/audit.log | grep nginx | grep denied | 
audit2allow -M mynginx
sudo semodule -i mynginx.pp

Basically you can check the permissions set on setsebool and correlate that with the error obtained from grepp'ing' audit.log nginx, denied

Answer 5

if "502 Bad Gateway" error throws on centos api url for api gateway proxy pass on nginx , run following command to solve the issue

sudo setsebool -P httpd_can_network_connect 1

Answer 6

1. Check the user in /etc/nginx/nginx.conf
2. Change ownership to user.

sudo chown -R nginx:nginx /var/lib/nginx

Answer 7

1. For first see what is denied:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied

type=AVC msg=audit(1618940614.934:38415): avc:  denied  { connectto } for
pid=18016 comm="nginx" path="/home/deployer/project/tmp/sockets/puma.sock" scontext=system_u:system_r:httpd_t:s0
tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
tclass=unix_stream_socket permissive=1

2. In my case it helps on CentOS7:

sudo setenforce 0

setsebool httpd_can_network_connect on -P
setsebool httpd_can_network_relay on -P

After you can see what is enable:

getsebool -a | grep httpd

httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> on
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_dbus_sssd --> off
httpd_dontaudit_search_dirs --> off
httpd_enable_cgi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_graceful_shutdown --> on
httpd_manage_ipa --> off
httpd_mod_auth_ntlm_winbind --> off
httpd_mod_auth_pam --> off
httpd_read_user_content --> off
httpd_run_ipa --> off
httpd_run_preupgrade --> off
httpd_run_stickshift --> off
httpd_serve_cobbler_files --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_sys_script_anon_write --> off
httpd_tmp_exec --> off
httpd_tty_comm --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_fusefs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
httpd_use_openstack --> off
httpd_use_sasl --> off
httpd_verify_dns --> off

Answer 8

I had the same issue. I tried @joebarbere solution of running the below command which fixed the issue but I was concerned about security

setsebool -P httpd_can_network_relay 1

I then came across the following solution

https://serverfault.com/questions/634294/nodejs-nginx-error-13-permission-denied-while-connecting-to-upstream

I was using non standard port and SELinux was blocking the port. Use the following command to check if the port is allowed

sudo semanage port --list | grep 4343

And added the port to allowed list

sudo semanage port --add --type http_port_t --proto tcp 4343

I restarted the nginx service, then the url was accessible. I would assume this is more secure.

Answer 9

13-permission-denied-while-connecting-to-upstreamnginx on centos server -

setsebool -P httpd_can_network_connect 1

Answer 10

I’ve run into this problem too. I'm using Nginx with HHVM, below solution fixed my issue:

sudo semanage fcontext -a -t httpd_sys_rw_content_t "/etc/nginx/fastcgi_temp(/.*)?"

sudo restorecon -R -v /etc/nginx/fastcgi_temp

Answer 11

Another reason could be; you are accessing your application through nginx using proxy but you did not add gunicorn.sock file for proxy with gunicorn.

You need to add a proxy file path in nginx configuration.

location / {
        include proxy_params;
        proxy_pass http://unix:/home/username/myproject/gunicorn.sock;
    }

Here is a nice tutorial with step by step implementation of this

https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04#configure-nginx-to-proxy-pass-to-gunicorn

Note: if you did not created anyname.sock file you have to create if first, either use above or any other method or tutorial to create it.

Answer 12

The SELinux Policy management tool has a set of standard ports that are allowed.
Usually 8001, 5000, 4000, etc are not one of them.

Step 1

check if the port you are trying to use is allowed (or assigned to something else)

sudo semanage port --list | grep http_port_t 

Output

http_port_t  tcp  80, 81, 443, 488, 8008, 8009, 8443, 9000

You can also confirm its use (assigned to something else)

sudo semanage port --list | grep 8001

Step 2

If it is not available then you can either add it or use one of the above options

Option 1

To add the port.

sudo semanage port --add --type http_port_t --proto tcp 8001

Option 2 (I recommend)

To use one of the allowed ports (instead of adding a new one)

server {
    ....

    location / {
        proxy_pass http://127.0.0.1:9000;
        ...
    }
}

In our case, we have changed 8001 to 9000.

Note: you should also change the port on your project to 9000
For this case is gunicorn mysite.wsgi:application --bind=127.0.0.1:9000

Security Implication

I consider this more secure than the previous answers

Answer 13

In my case, it was fixed (from this blog) by running the command: semanage permissive -a httpd_t and restarting NGINX service.

Answer 14

I encountered a similar issue while deploying my Django application locally with Nginx as a reverse proxy. The error message (13: Permission denied) while connecting to upstream typically indicates a permission problem with accessing the upstream server, in this case, your Django app.

The solution that worked for me was related to the nginx.conf file. Inside this file, the user directive plays a significant role, as it defines the user and group that Nginx runs as. Sometimes, a comment preceding the user directive can cause permission issues.

To resolve the error, follow these steps:

1. Locate your nginx.conf file (usually in /etc/nginx/).
2. Find the user directive. It should look like: user <user>;.
3. Make sure there are no comments (lines starting with #) before the user directive. Comments can sometimes disrupt the configuration parsing.
4. Save the nginx.conf file.
5. Restart Nginx with: sudo systemctl restart nginx.

After applying these steps, the (13: Permission denied) error should no longer occur, and your Django application should function properly when accessed through the Nginx proxy.

Answer 15

Do not install SELinux on Ubuntun 22.10. I did that, and I could no longer boot. Ubuntun 22.10 gave me the GRUB menu, I don't know how to recover: I ended up reinstalled Ubuntun 22.10.