413 error with Kubernetes and Nginx ingress controller

Question

I'm trying to change the client_max_body_size value, so my NGINX ingress will not return the HTTP 413 Content Too Large error (as seen in the logs).

I've tested a few solutions.
Here is my config map:

kind: ConfigMap
apiVersion: v1
data:
  proxy-connect-timeout: "15"
  proxy-read-timeout: "600"
  proxy-send-timeout: "600"
  proxy-body-size: "8m"
  hsts-include-subdomains: "false"
  body-size: "64m"
  server-name-hash-bucket-size: "256"
  client-max-body-size: "50m"
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx

These changes had no effect at all: in NGINX controller's log I can see the information about reloading the config map, but the values in nginx.conf are the same:

$ cat /etc/nginx/nginx.conf | grep client_max                                                                                                       
                            client_max_body_size                    "8m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";

My nginx-controller config uses this image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.13.0

How can I force NGINX to change this setting? I need to change it globally, for all my ingresses.

score 102 · Accepted Answer · edited Apr 26 '18 at 11:39

102

You can use the annotation nginx.ingress.kubernetes.io/proxy-body-size to set the max-body-size option right in your Ingress object instead of changing a base ConfigMap.

Here is the example of usage:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
...

edited Apr 26 '18 at 11:39

Adrián

2,876
17
22

answered Apr 19 '18 at 10:12

Anton Kostenko

8,200
2
30
37

2

I wanted to set it globally, but actually this approach is better as I have my yaml files within the project repository. So it won't be lost. Thanks. – Djent Apr 19 '18 at 11:13
1

So the `client-max-body-size` config line can be removed entirely from the `metadata.annotations` ? – payne Jul 07 '21 at 20:32
2

@payne it depends. refer to https://stackoverflow.com/a/67923692/1086907. If the body is too big, `client-max-body-size` is required to improve the performance. See [Annotations: Client Body Buffer Size](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-body-buffer-size) – Peter Sep 10 '21 at 07:08
`kubectl patch ingress my-ingress -p '{"metadata": {"annotations": {"nginx.ingress.kubernetes.io/proxy-body-size": "0"}}}'` – kinjelom Jun 07 '22 at 07:33
For me it was `nginx.ingress.kubernetes.io/client-body-buffer-size` – leonp5 Jan 03 '23 at 13:41

score 18 · Answer 2 · answered Aug 30 '22 at 21:49

There is confusion here about whether the correct annotation is nginx.ingress.kubernetes.io/proxy-body-size or nginx.org/client-max-body-size. The answer is that it might be either!

In the unassailable wisdom of everyone involved, there are two different "NGINX ingresses" in the world. It's hard to tell which you are running, and Google Search is useless at differentiating here (but what's new).

If you are using the official NGINX Ingress, the correct annotation is here: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

If you are using the official Kubernetes Ingress based on NGINX, the correct annotation is this: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-max-body-size

If the Docker image name of your ingress implementation is k8s.gcr.io/ingress-nginx/controller then you're using the official Kubernetes version. By elimination, you can determine if you're using the NGINX implementation.

I hope everyone who contributed to this confusing state has a very bad day. The rest of you... stay cheery!

The part about figuring out which version I had running finally helped me! — zeisi, Nov 02 '22 at 12:29
More info and comparison of two Nginx ingresses can be found in [Nginx blog](https://www.nginx.com/blog/guide-to-choosing-ingress-controller-part-4-nginx-ingress-controller-options/). — izogfif, Feb 20 '23 at 06:27

score 16 · Answer 3 · answered Sep 02 '18 at 20:57

16

To set it globally, this configmap.md documentation might be helpful. Turns out the variable to set is proxy-body-size, not client-max-body-size.

When you deploy the helm chart, you can set --set-string controller.config.proxy-body-size="4m".

answered Sep 02 '18 at 20:57

SoJeN

429
5
7

score 7 · Answer 4 · answered Jun 10 '21 at 14:55

7

Maybe some people got stuck here recently. We just made out that the annotation nginx.ingress.kubernetes.io/proxy-body-size is not longer being applied and the correct one would be:

  annotations:
    nginx.org/client-max-body-size: "999m"

Hope this helps to others.

answered Jun 10 '21 at 14:55

Edorka

1,781
12
24

1

By the way, those two annotations for two different nginx ingress controllers. The domain in the prefix is a good reference. – Draculater Dec 22 '21 at 03:21
@Draculater indeed. The problem is that google search sends people to this post and I thought (maybe wrong) that, however this is not the correct question, this response can shorten their search for a solution. – Edorka Dec 30 '21 at 08:18

score 6 · Answer 5 · answered Jan 29 '20 at 16:22

6

Update:

I have been experiencing the same problem and no solutions were working. After reading through countless blogs and docs that all had the same suggested solution I found that they have changed the naming convention.

It is no longer denoted by "proxy-body-size" or this just never works for me.

link below shows that the correct configmap variable to use is "client-max-body-size"

https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/

answered Jan 29 '20 at 16:22

Joe Roberts

99
1
6

9

For those reading, this comment applies when using the NGINX-issued ingress controller. If you are using the Kubernetes-issued NGINX controller the property name is still `proxy-body-size`. You can tell this by the annotation key which will be `nginx.ingress.kubernetes.io/xxxxx` for Kubernetes-issued rather than `nginx.org/xxxxx` which is NGINX-issued. – James G Oct 12 '20 at 05:07
6

I encourage the folks at Nginx and Kubernetes to get together and develop a common spec – code_monk Nov 24 '20 at 03:14

mati kepa · Answer 6 · 2021-07-26T08:29:41.227

2

you can always make a global change and disable it at all

kubectl -n ingress-nginx patch configmap \
nginx-configuration -p '{"data": {"proxy-body-size": "0"}}'

or set to 5tb

kubectl -n ingress-nginx patch configmap \
nginx-configuration -p '{"data": {"proxy-body-size": "5tb"}}'

edited Jul 26 '21 at 08:29

answered May 19 '21 at 16:25

mati kepa

2,543
19
24

Hurell Lyons · Answer 7 · 2020-03-10T20:56:47.930

I have tried both proxy-body-size and client-max-body-size on the configmap and did a rolling restart of the nginx controller pods and when I grep the nginx.conf file in the pod it returns the default 1m. I am trying to do this within Azure Kubernetes Service (AKS). I'm working with someone from their support. They said its not on them since it appears to be a nginx config issue.

The weird hting is we had other clusters in Azure that this wasnt an issue on until we discovered this with some of the newer deployments. The initial fix they came up with is what is in this thread but it just refuses to change.

Below is my configmap:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  client-max-body-size: 0m
  proxy-connect-timeout: 10s
  proxy-read-timeout: 10s
kind: ConfigMap
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"nginx-nginx-ingress-controller-7b9bff87b8-vxv8q","leaseDurationSeconds":30,"acquireTime":"2020-03-10T20:52:06Z","renewTime":"2020-03-10T20:53:21Z","leaderTransitions":1}'
  creationTimestamp: "2020-03-10T18:34:01Z"
  name: ingress-controller-leader-nginx
  namespace: ingress-nginx
  resourceVersion: "23928"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader-nginx
  uid: b68a2143-62fd-11ea-ab45-d67902848a80

After issuing a rolling restart: kubectl rollout restart deployment/nginx-nginx-ingress-controller -n ingress-nginx

Grepping the nginx ingress controller pod to query the value now reveals:

kubectl exec -n ingress-nginx nginx-nginx-ingress-controller-7b9bff87b8-p4ppw cat nginx.conf | grep client_max_body_size
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    21m;

Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes.

I can confirm this, we are running in exactly the same problem. AKS version 1.18.4. — Patrick P., Aug 25 '20 at 15:12
It seems adding `nginx.ingress.kubernetes.io/proxy-body-size: "50m"` annotation on ingress updates the "client_max_body_size" value on the nginx controller config file. — Somnath Pawar, May 13 '22 at 09:46

score 0 · Answer 8 · edited Aug 15 '20 at 13:08

0

I am able to fix it by redeploying the nginx-ingress controller after modifying the configmap.

kubectl patch deployment your_deployment -p "{\"spec\": {\"template\": {\"metadata\": { \"labels\": {  \"redeploy\": \"$(date +%s)\"}}}}}"

edited Aug 15 '20 at 13:08

Arghya Sadhu

41,002
9
78
107

answered Aug 15 '20 at 12:51

Selvam Elangovan

19
1

413 error with Kubernetes and Nginx ingress controller

8 Answers8

Linked