Cloud Run: Why does python flask return 400 Bad request? But locally everything is fine

Question

I have a simple flask application. And I need to run it on Cloud Run with enabled option "Manage authorized users with Cloud IAM."

app.py

from flask import Flask

api_app = Flask(__name__)

endpoints.py

from app import api_app

@api_app.route("/create", methods=["POST"])
def api_create():
    # logic

main.py

from app import api_app
from endpoints import *    


if __name__ == "__main__":
    api_app.run(host="0.0.0.0", port=8080)

And it works if i run it locally. It also works well if run in docker. When I upload the application image to Cloud Run, there are no deployment errors. But when I try to call the endpoint I get an error even though everything is working fine locally.

Request example:

import urllib
import google.auth.transport.requests
import google.oauth2.id_token
import requests
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "test.json"

audience="https://test-service-*****.a.run.app"

req = urllib.request.Request(audience)
auth_req = google.auth.transport.requests.Request()
token = google.oauth2.id_token.fetch_id_token(auth_req, audience)

response = requests.post(f"{audience}/create", data={
    "text": "cool text"
}, headers={
    "Authorization": f"Bearer {token}"
})

Response:

<!doctype html>
<html lang=en>
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>

In Cloud Run logs I have only one warning log POST 400 820b 3ms python-requests/2.26.0 https://test-service-*****.a.run.app/create

Dockerfile:

FROM python:3.8-slim-buster

WORKDIR /alpha
ENV PYTHONUNBUFFERED True

COPY . .

RUN pip install --no-cache-dir -r requirements.txt

CMD exec gunicorn --bind :8080 --workers 1 --threads 8 --timeout 0 main:api_app

Why can't I reach the application endpoint in Cloud Run?

You should really look at the [Cloud Run logs](https://cloud.google.com/run/docs/logging) and see what they say... — Edo Akse, Nov 15 '22 at 19:02
@EdoAkse I looked. Error 400 duplicated in logs. Nothing to indicate an error — Devid Mercer, Nov 15 '22 at 19:04
@EdoAkse no errors. As I said, the same version works fine locally, but I can't access the endpoints in Cloud Run for the same version. — Devid Mercer, Nov 15 '22 at 19:13
What's the `endpoint` variable value? Shouldn't be audience? — guillaume blaquiere, Nov 16 '22 at 08:58
There is no need to set "GOOGLE_APPLICATION_CREDENTIALS" and include a JSON service account key in your container. Instead, just use the --service-account option when you deploy your Cloud Run service with "gcloud run deploy". One less thing that can go wrong, and your container won't include a dangerous service account key. — Martin Omander, Nov 21 '22 at 23:57

Edo Akse · Accepted Answer · 2022-11-16T19:20:15.773

1

So I did some testing, and the only thing I did was remove the port binding from the Dockerfile CMD exec gunicorn and from the main.py. Note that the dundermain thingy is not needed as gunicorn takes care of that.

After that it worked as expected.

Note that I did not set it up as a private endpoint as I was to lazy to jump through the hoops for that, and the response code you get back is not a 403.

Note that I also made sure that /create returned something. I'm going to assume that you did the same.

EDIT

I've made the CR endpoint private, and created a serviceaccount with the role "Cloud Run Invoker", created a JSON key and saved it as test.json

I've also added some logic to print out the token, and to decode the first part. Your output should look similar to mine.

EDIT 2

In the end it ended up Flask not being able to deal with the incoming data. There's a very nice answer here which explains how to get the data from all the different ways it can get POSTed

FROM python:3.8-slim-buster

WORKDIR /alpha
ENV PYTHONUNBUFFERED True

COPY . .

RUN pip install --no-cache-dir -r requirements.txt

CMD exec gunicorn  --workers 1 --threads 8 --timeout 0 main:api_app

endpoints.py

from app import api_app
from flask import request


@api_app.route("/create", methods=["POST"])
def api_create():
    data = request.data
    return f"{data}"

main.py

from app import api_app
from endpoints import *

perform_request.py

import urllib
import requests
import google.auth.transport.requests
import google.oauth2.id_token
import os
import base64


os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "test.json"
target_uri = "https://pasta*****.a.run.app"


req = urllib.request.Request(target_uri)
auth_req = google.auth.transport.requests.Request()
token = google.oauth2.id_token.fetch_id_token(auth_req, target_uri)

print(token, "\n")
print(base64.b64decode(f"""{token.split(".")[0]}==""".encode("utf-8")), "\n")


response = requests.post(
    f"{target_uri}/create",
    data={"text": "cool text"},
    headers={"Authorization": f"Bearer {token}"},
)

print(response.text)

output

eyJhbGciOiJSUzI1NiIsIm[SNIPPED]

b'{"alg":"RS256","kid":"713fd68c9[SNIPPED]","typ":"JWT"}' 

b'text=cool+text'

edited Nov 16 '22 at 19:20

answered Nov 15 '22 at 19:47

Edo Akse

4,051
2
10
21

Alas, I see the same error. I also changed dockerfile to ```CMD exec gunicorn --bind :443 --workers 1 --threads 8 --timeout 0 main:api_app``` – Devid Mercer Nov 15 '22 at 20:01
are you actually using waitress in your deployed app? – Edo Akse Nov 15 '22 at 20:20
Cloud Run does not support applications that listen on anything but HTTP. Your suggestion will **not** work for a Cloud Run application. The Cloud Run Frontend (GFE) proxies public HTTPS to internal HTTP. SSL is offloaded. – John Hanley Nov 15 '22 at 20:35
@EdoAkse yes, I use waitress in deployed app – Devid Mercer Nov 15 '22 at 20:58
1

you know both `gunicorn` and `waitress` are WSGI servers? You're using WSGI server to run another WSGI server. I would definitely remove `waitress` from your code, and just use `gunicorn` in your Dockerfile – Edo Akse Nov 15 '22 at 21:01
@EdoAkse Yes, I know. I have already tried to use ```waitress``` only - with no result. I have only used ```gunicorn``` with no results. Using them together didn't change anything either. Anyway I see the same error. – Devid Mercer Nov 15 '22 at 21:10
@EdoAkse I clarified the question, maybe I missed important details trying to simplify the example. Thanks for your time and willingness to help – Devid Mercer Nov 16 '22 at 09:29
1

Check my updated answer, it's simpler than you think – Edo Akse Nov 16 '22 at 18:05
@EdoAkse Alas. The error seems to be related to the way requests are sent, and not to the service itself. – Devid Mercer Nov 16 '22 at 18:20
1

let me try again on a private endpoint then – Edo Akse Nov 16 '22 at 18:23
@EdoAkse I did some tests. `GET` requests work fine with "Cloud IAM" enabled and when it's disabled. The problem is only in the `POST`. Maybe something is wrong with the headers... I continue to test – Devid Mercer Nov 16 '22 at 18:57
1

so, either there's something wrong with the token, or with the actual code that is in the `/create` endpoint. You can try to create a simplistic endpoint just as I did to determine if it's the code running in it – Edo Akse Nov 16 '22 at 18:58
@EdoAkse Thanks for the good example. I completed it using `Content-type: application/json` in requests and it helped me solve the problem. Thanks again for your help and time – Devid Mercer Nov 16 '22 at 19:13
1

Please be aware that there are a lot of ways to pass data with a POST. There's a SO post with 23 answers on how to get data in Flask, all valid. [Linky](https://stackoverflow.com/questions/10434599/get-the-data-received-in-a-flask-request) – Edo Akse Nov 16 '22 at 19:22

Cloud Run: Why does python flask return 400 Bad request? But locally everything is fine

1 Answers1