11

If I curl a POST request with file upload to my google compute load balancer (LB) I get a 502 error. If I do the same curl to the worker node behind the LB, it works. If I use a library like PHP Guzzle, it works either way. If I do a basic GET request on the LB, I get the correct response but the worker log does not acknowledge receiving the request as if the LB cached it. What is going on? FYI, google LB newb. Thanks

Edit:

I'm using GCE HTTP LB. The Curl command looks like this:

curl http://1.2.3.4 -F "key=value" -F "data=@path/to/file"

This curl command works when using the GCE VM IP but does NOT when using the GCE HTTP LB IP.

Matthew Scragg
  • 4,540
  • 3
  • 19
  • 27

2 Answers2

6

This one line of code fixed it for me:

curl_setopt($ch, CURLOPT_HTTPHEADER, ['Expect:']);

You obviously need to add the empty Expect: header to whatever other headers you're sending up, but that header is what fixes cURL for use with Google HTTP load balancers.

More Info

The Google document Setting Up HTTP(S) Load Balancing has a note near the bottom in the Notes and Restrictions section saying that HTTP/1.1 100 Continue responses are not supported.

It seems that by default cURL will always set Expect: 100-continue headers when you send a POST request. Thus apparently cURL is unable to send POST through a GCE HTTP load balancer by default.

On the end-user side, you just see 502 responses coming back from Google, which is all the more confusing since making the exact same POST to a server that is not behind the load balancer works perfectly fine.

However, presence of Expect: 100-continue causes the Google Load Balancer to freak out and break the request.

On the server side the POST data cannot be parsed (it does not even arrive to the server, though the Content-Length is reported correctly). In my case this caused the server to return a 500 Internal Server Error, which the GCE LB munges and sends back to the user as a 502 Bad Gateway error.

After adding an empty Expect: header, my POST data is making it to my load balanced VMs correctly, they're parsing and returning valid responses, and my client is getting a 200 instead of a 502.

Thanks to this question which helped shed some light on the issue.

Community
  • 1
  • 1
Ross Perkins
  • 161
  • 1
  • 4
  • 2
    Damn it, Ross, if I could give you a trillion votes for this, I would! It's one of those one-liner week-long head-banging hair-pulling issues. – Zlatko Dec 08 '16 at 17:16
  • Btw, you know how to unset this or set to empty by default for command-line curl? – Zlatko Dec 08 '16 at 17:22
  • @Zlatko I think you just need to add an empty header in the command line version as well. I'm not super familiar with curl command line but I'd imagine it works the same as the lib. – Ross Perkins Dec 09 '16 at 17:59
  • Yes, I do that. But I wanted some environment or whatever setting for this, because I'm sure I'll forget all about this next time I get the errors. – Zlatko Dec 09 '16 at 20:32
1

Traffic from the load balancer to your instance is not enabled by default. Unfortunately this is not well documented, and really, when you create a load balancer this should happen automatically.

Try adding this firewall rule the network that your load balancer and VMs are on:

130.211.0.0/22   tcp:1-5000   Apply to all targets
Tyguy7
  • 573
  • 8
  • 15
  • Indeed, it is most likely a firewall issue which is not specified by default. The Load Balancer and the GCE instances are on different networks that can not talk by default. – Jason Mayes Apr 28 '16 at 22:56