6

I have a node app bundled with webpack which I am trying to deploy to cloud run. In order to be able to load a file more than 32 MB (the set limit) in cloud run I added the header ('Transfer-encoding' : 'chunked') to the webpack config file. (The file is a min.js file ,the minified version of the website and is loaded upon hitting the url).

Locally everything runs fine but when this change is deployed on cloud run, it gives the follwoing error upon clicking on the app url after it is deployed:

(502 error)

"upstream connect error or disconnect/reset before headers. reset reason: protocol error"

I have disabled the http2 end-to-end option in deployment.

Here is the webpack file where the header is added

webpack.config.js

devServer: {

    contentBase: path.resolve(__dirname, 'build'),

    host: '0.0.0.0',
    port: process.env.PORT || 8601,
    sockPort: 'location',
    disableHostCheck: true,
    devServer: {
       headers: {
         'Transfer-encoding': 'chunked',
       },
    },
    
}
Ryan M
  • 18,333
  • 31
  • 67
  • 74
Vibhor Gaur
  • 61
  • 1
  • 1
  • 5
  • From your desktop run the curl command with the -v option to your Cloud Run services. Post the output in your question (redact sensitive information): `curl -v https://example.com`. Replace `example.com` with your endpoint URL. – John Hanley Jun 13 '22 at 17:51
  • Hi Joh, actually, rather than setting the transfer encoding header (http1.1) now I am setting, (http2: true) in the devServer (as cloud run documentation says that with http2 there are no size limits in request and response, so no 32 mb limits). Again ran fine locally and with docker run (with https, unsecure), also saw significantly faster load times. But when deployed to Cloud run it gives 503, service unavailable. Again tried with both http2 flag enabled and disabled in the deploy. Posted the 503 curl output in the post above. – Vibhor Gaur Jun 13 '22 at 18:00
  • 503 means your service has crashed/failed. Look in the Cloud Run logs for your service for more details. Note: I do not see the output from `curl -v` in your question. – John Hanley Jun 13 '22 at 18:02
  • also, am using port 80 – Vibhor Gaur Jun 13 '22 at 18:07
  • added the curl output, also checking the cloud run server logs don't get any other information, they are similar to how it runs locally with the same end message ("compiled successfully") and no error logs, could the connection issue be due to using http2? – Vibhor Gaur Jun 13 '22 at 18:17
  • You said you are not using **http2**. What are you using port 80 for? Requests to Cloud Run must be made using HTTPS (port 443). Requests to port 80 will be redirected. – John Hanley Jun 13 '22 at 18:19
  • (only error logs are GET 503) – Vibhor Gaur Jun 13 '22 at 18:24
  • If there are no additional details in the Cloud Run logs for your service, then add logging to your application. – John Hanley Jun 13 '22 at 18:26
  • port 80 is the container port, am using http2 but have disabled the option in cloud run deploy (the option which says use "http2 end to end") – Vibhor Gaur Jun 13 '22 at 18:28
  • `reset reason: protocol error` - typically means the gRPC server closes the HTTP/2 stream. can you try to use "`gcloud beta deploy ... --use-http2`" to redeploy your service – Roopa M Sep 12 '22 at 12:00

3 Answers3

7

Root Cause

I had earlier checked the option Use HTTP/2 end-to-end that said Use if your container is a gRPC streaming server or is able to directly handle requests in HTTP/2 cleartext.

I did that for some troubleshooting purposes. It often takes some time to take those seetings into effect after you select Use HTTP/2 end-to-end and Deploy your service on Cloud Run.

So you might not see the failures upfront when you turn on Use HTTP/2 end-to-end. Failures such as gcp cloud run upstream connect error or disconnect/reset before headers. reset reason: protocol error might arise.

enter image description here

Fix

To fix, unselect from GCP Console, navigate to Cloud Run -> Your Service -> Edit -> Connections Tab -> Use HTTP/2 end-to-end.

enter image description here

Then choose the option Serve this revision immediately and click on Deploy.

enter image description here

Now, your application should work properly as expected as shown in my case.

enter image description here

navule
  • 3,212
  • 2
  • 36
  • 54
1
TLDR: Check your response header for special characters.

Hi,

we had the very same 502 response on calls to our Cloud Run instance, with the body:

upstream connect error or disconnect/reset before headers. reset reason: protocol error

Our Cloud Run service provides a document generation, where an attribute of the json input was used as part of the document name. Our response headers contained a header entry like this:

Content-Disposition: attachment; filename=<the-attribute-value>.pdf

Our service also runs in countries with the attribute written in cyrillic letters. This caused the issue for us.

We found an alternative for the filename, which was independent of the input of the user.

Hope that helps!

simonnorra
  • 11
  • 1
0

I actually encountered this exact error because I was sending a 204 response with a message body. Change the status to 200 or 202 if that's the case for you.

Kraigolas
  • 5,121
  • 3
  • 12
  • 37