15

When I try to access my website deployed on Google Cloud I receive the following error page:

We're sorry...

... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.

Here are the details of my setup:

  • NextJS web application deployed to Cloud Run (traffic limited to Internal + Load Balancing)
  • Strapi Content Management System deployed to Cloud Run (traffic limited to Internal + Load Balancing)
  • I have a load balancer in front of each Cloud Run service and am using Google reserved IP's and managed SSL certs
  • Identity Aware Proxy (IAM) is in front of both Load Balancers
  • Each page visit makes an API call to the content management system to retrieve content (could this be causing the network to think "automated queries" are occurring?)

Troubleshooting:

  • It doesn't appear to be related to the actual user visiting the website. This error message appears for all users, different devices, different IP addresses. It was working without issue before, this is the first time I've had this problem.
  • No traffic is reaching the Cloud Run environments. If I switch their ingress to Allow All I am able to hit the Cloud Run URL directly and it works
  • Deleting and recreating the load balancer doesn't resolve the issue

Based on this, there seems to be some network configuration/security occurring near the load balancer.

Any solutions or suggestions to continue debugging?

David Beaudway
  • 794
  • 10
  • 27
  • 2
    +1 The issue looks like related only when we try to access the cloud run using a NGINX proxy pass in our case from a different domain using an internal load balancer. I thought it may be related to the Host header? but the weird fact that it started to fail overnight is weird. – Tonino Jan 12 '23 at 10:57
  • Has anyone had any luck tweaking the passed Nginx headers to see if things are ok then? – Filip Dupanović Jan 12 '23 at 23:41
  • I tried to log the header passed with and without NGINX and IAP Load Balancer to the Cloud RUN. The main difference are in the Host, cookies and the x-goog (coming from iap),x-forwarded-for headers. Modifying the host header in nginx doesn't make any difference but because the issue is related to IAP, the only thing we are trying to change are the x-goog headers and the cookies. I will keep you posted – Tonino Jan 13 '23 at 09:53
  • 1
    @FilipDupanović I was able to make NGINX work with IAP proxying the request to a Cloud RUN removing the header related to the jwt token. We don't do validation of that JWT token as that cloud run is a public API so we don't care about that header. Check it out if that works for you as well, I will update my answer with the nginx config – Tonino Jan 13 '23 at 12:18

1 Answers1

5

So far the two only workarounds available are:

Both are not acceptable on production systems unless you don't care about security. We are trying to work out what are the IAP headers lost between NGINX and Cloud RUN but I guess the issue affects also infrastructure without NGINX

EDIT the issue tracker post https://issuetracker.google.com/issues/265184232 looks related to the issue. The workaround suggested by the google team is to switch to a Global HTTP Classic Load Balancer We tried and it did not work but we may have a different problem. Hope that helps anyone else for the time being

EDIT 2 In our NGINX configuration we proxy the request to a public Cloud RUN API which at the moment is affected by the issue.

I was able to proxy the request successfully just by removing the JWT header added by IAP (leaving any other headers with the authenticated user) example of my config

 location /api/ {
    #remove iap jwt token header 
    proxy_set_header        X-Goog-Iap-Jwt-Assertion "";

    # proxy to the public cloud run instance
    proxy_pass https://myCloudRun-123456basd-ew.a.run.app/;
}
Tonino
  • 1,137
  • 10
  • 25
  • 1
    Dropping `X-Goog-Iap-Jwt-Assertion` with [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) worked for me, thank you! – Filip Dupanović Jan 13 '23 at 20:28