Argo and Kubernetes "Request entity to large: limit is 3145728"

Question

I've been trying to deploy a workflow in Argo with Kubernetes and I'm getting this error

Can someone help me to know the root of the issue?

I’ve tried several things but I’ve been unsuccessful.

Answer 1

The way Argo solves that problem is by using compression on the stored entity, but the real question is whether you have to have all 3MB worth of that data at once, or if it is merely more convenient for you and they could be decomposed into separate objects with relationships between each other. The kubernetes API is not a blob storage, and shouldn't be treated as one.

• The "error": "Request entity too large: limit is 3145728" is probably the default response from kubernetes handler for objects larger than 3MB, as you can see here at L305 of the source code:

expectedMsgFor1MB := etcdserver: request is too large expectedMsgFor2MB := rpc error: code = ResourceExhausted desc = trying to send message larger than max expectedMsgFor3MB := Request entity too large: limit is 3145728 expectedMsgForLargeAnnotation := metadata.annotations: Too long: must have at most 262144 bytes

• The ETCD has indeed a 1.5MB limit for processing a file and you will find on ETCD Documentation a suggestion to try the--max-request-bytes flag but it would have no effect on a GKE cluster because you don't have such permission on master node.

But even if you did, it would not be ideal because usually this error means that you are consuming the objects instead of referencing them which would degrade your performance.

I highly recommend that you consider instead these options:

- Determine whether your object includes references that aren't used

- Break up your resource

- Consider a volume mount instead

There's a request for a new API Resource: File (orBinaryData) that could apply to your case. It's very fresh but it's good to keep an eye on.

Partial source for this answer: https://stackoverflow.com/a/60492986/12153576