Google Storage API Forbidden ipRefererBlocked Error

Question

I'm getting the following error when using the Google Storage JSON API:

$ curl -v 'https://www.googleapis.com/storage/v1/b/MY-BUCKET/o?key=MY_API_KEY'
* Hostname was NOT found in DNS cache
*   Trying 64.233.186.95...
* Connected to www.googleapis.com (64.233.186.95) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: *.storage.googleapis.com
* Server certificate: Google Internet Authority G2
* Server certificate: GeoTrust Global CA
> GET /storage/v1/b/MY-BUCKET/o?key=MY_API_KEY HTTP/1.1
> User-Agent: curl/7.37.1
> Host: www.googleapis.com
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< Vary: X-Origin
< Content-Type: application/json; charset=UTF-8
< Date: Tue, 31 Mar 2015 16:57:40 GMT
< Expires: Tue, 31 Mar 2015 16:57:40 GMT
< Cache-Control: private, max-age=0
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
* Server GSE is not blacklisted
< Server: GSE
< Alternate-Protocol: 443:quic,p=0.5
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
< 
{
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "ipRefererBlocked",
    "message": "There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed.",
    "extendedHelp": "https://console.developers.google.com"
   }
  ],
  "code": 403,
  "message": "There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed."
 }
}

But everything seems configured correctly. The key I'm using is listed as allowing any IP:

The APIs themselves are active:

And the project has billing information correctly configured:

I've tried enabling and disabling the APIs and refreshing the billing info. Nothing is helping. What are the next steps? Try AWS?

Answer 1

For posterity here is what happened:

I spun a different VM in the same datacenter (Linode in Newark/NJ) and it worked perfectly, so the problem seemed not be my key, but my IP itself.

In order to get to the bottom of this I had to pay Google Cloud US$ 150.00 so they could let me open a support ticket and describe the issue.

I spend the next two days arguing with a support staffer that – while being very polite – always assumed the error was in my end instead of theirs.

After writing a 6000 character reply with super specific details he finally decided to check with someone and found out my IP had been blocked by a Google traffic automation system because it erroneously detected that my IP came from a restricted country (Iran).

Given that the VM was in New Jersey – and it was very easy to see that through traceroute – they told me they would have to manually overwrite the system and place my IP back in the US. This would take three days, so I waited.

Three days later I got an email telling me to test everything, and so I did. It didn't work. After another couple of emails and waiting another day they finally fixed it.

The cherry on top of this whole mess was having a Google Support staffer accuse my company of doing business with restricted countries and acting as a proxy to traffic coming from Iran. That was preposterous and it really offended us, because even after making a mistake they still wanted to blame us for doing nothing wrong.

We told two engineers to turn that VM backwards and see if they could find any evidence of that it had been hacked and of course they couldn't. It was a fully patched Ubuntu 14.04 server that one could only SSH through ones private key and remote root login was disabled.

And this is how the tale ends, folks. Spending US$ 150.00 to notify Google of their own mistake and having to fight your way to the solution.

Oh, and one more thing: we're migrating to AWS.