2

I'm trying to use the Google Indexing API using the PHP client library.

This is my code:

    $client = new Google_Client();

    //use the private key that we created for our service account.
    $client->setAuthConfig(storage_path('google_auth_config.json'));    //this works
    $client->addScope('https://www.googleapis.com/auth/indexing');

    // Get a Guzzle HTTP Client
    $httpClient = $client->authorize();
    $endpoint = 'https://indexing.googleapis.com/v3/urlNotifications:publish';

    $content = '{
      "url": "https://myverifieddomain.com/url",
      "type": "URL_UPDATED"
    }';

    $response = $httpClient->post($endpoint, [ 'body' => $content ]);
    $status_code = $response->getStatusCode();

The auth part works.

However, the $status_code I get is 403. Here's the full $response:

object(GuzzleHttp\Psr7\Response)#2233 (6) {
  ["reasonPhrase":"GuzzleHttp\Psr7\Response":private]=>
  string(9) "Forbidden"
  ["statusCode":"GuzzleHttp\Psr7\Response":private]=>
  int(403)
  ["headers":"GuzzleHttp\Psr7\Response":private]=>
  array(11) {
    ["Vary"]=>
    array(3) {
      [0]=>
      string(8) "X-Origin"
      [1]=>
      string(7) "Referer"
      [2]=>
      string(22) "Origin,Accept-Encoding"
    }
    ["Content-Type"]=>
    array(1) {
      [0]=>
      string(31) "application/json; charset=UTF-8"
    }
    ["Date"]=>
    array(1) {
      [0]=>
      string(29) "Fri, 24 Jun 2022 10:26:02 GMT"
    }
    ["Server"]=>
    array(1) {
      [0]=>
      string(3) "ESF"
    }
    ["Cache-Control"]=>
    array(1) {
      [0]=>
      string(7) "private"
    }
    ["X-XSS-Protection"]=>
    array(1) {
      [0]=>
      string(1) "0"
    }
    ["X-Frame-Options"]=>
    array(1) {
      [0]=>
      string(10) "SAMEORIGIN"
    }
    ["X-Content-Type-Options"]=>
    array(1) {
      [0]=>
      string(7) "nosniff"
    }
    ["Alt-Svc"]=>
    array(1) {
      [0]=>
      string(162) "h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43""
    }
    ["Accept-Ranges"]=>
    array(1) {
      [0]=>
      string(4) "none"
    }
    ["Transfer-Encoding"]=>
    array(1) {
      [0]=>
      string(7) "chunked"
    }
  }
  ["headerNames":"GuzzleHttp\Psr7\Response":private]=>
  array(11) {
    ["vary"]=>
    string(4) "Vary"
    ["content-type"]=>
    string(12) "Content-Type"
    ["date"]=>
    string(4) "Date"
    ["server"]=>
    string(6) "Server"
    ["cache-control"]=>
    string(13) "Cache-Control"
    ["x-xss-protection"]=>
    string(16) "X-XSS-Protection"
    ["x-frame-options"]=>
    string(15) "X-Frame-Options"
    ["x-content-type-options"]=>
    string(22) "X-Content-Type-Options"
    ["alt-svc"]=>
    string(7) "Alt-Svc"
    ["accept-ranges"]=>
    string(13) "Accept-Ranges"
    ["transfer-encoding"]=>
    string(17) "Transfer-Encoding"
  }
  ["protocol":"GuzzleHttp\Psr7\Response":private]=>
  string(3) "1.1"
  ["stream":"GuzzleHttp\Psr7\Response":private]=>
  object(GuzzleHttp\Psr7\Stream)#2221 (7) {
    ["stream":"GuzzleHttp\Psr7\Stream":private]=>
    resource(767) of type (stream)
    ["size":"GuzzleHttp\Psr7\Stream":private]=>
    NULL
    ["seekable":"GuzzleHttp\Psr7\Stream":private]=>
    bool(true)
    ["readable":"GuzzleHttp\Psr7\Stream":private]=>
    bool(true)
    ["writable":"GuzzleHttp\Psr7\Stream":private]=>
    bool(true)
    ["uri":"GuzzleHttp\Psr7\Stream":private]=>
    string(10) "php://temp"
    ["customMetadata":"GuzzleHttp\Psr7\Stream":private]=>
    array(0) {
    }
  }
}

I've enabled the API as instructed here.

The service account is added as an owner as instructed here.

I have no idea what else I can do. The error message doesn't appear to have any additional information.

General Grievance
  • 4,555
  • 31
  • 31
  • 45
sveti petar
  • 3,637
  • 13
  • 67
  • 144
  • Forbidden means you dont have access – Linda Lawton - DaImTo Jun 25 '22 at 15:00
  • The $httpClient looks fine when I var_dump it. Could the 403 be related to this endpoint specifically? – sveti petar Jun 27 '22 at 08:30
  • Conduct or not, I talk here the way developers talk to each other in real life :) At least it's honest. – sveti petar Jun 27 '22 at 08:31
  • what exactly is storage_path are you sure this isnt an issue with accessing google storage? – Linda Lawton - DaImTo Jun 27 '22 at 11:01
  • 1
    No, it's a local Laravel storage path where I keep the JSON file I received from Google for auth. I've confirmed that this part works. – sveti petar Jun 27 '22 at 11:06
  • Can you post the full error message. See my answer. There are serval possible causes for your forbiden error. – Linda Lawton - DaImTo Jun 27 '22 at 11:40
  • That empty GuzzleHttp response doesn't provide any information, and there are quite a few few [possible causes](https://developers.google.com/search/apis/indexing-api/v3/core-errors#FORBIDDEN) for HTTP 403 (forbidden still hints for permissions). What happens when performing a GET on `https://indexing.googleapis.com/v3/urlNotifications/metadata?url=url-encoded_url`?? – Martin Zeitler Jun 27 '22 at 12:46
  • Checking that you have referred to the Google Docs (and not just Stack Overflow answers / PHP SDK) https://developers.google.com/search/apis/indexing-api/v3/prereqs – Robbie Jun 29 '22 at 06:15
  • @Robbie Yes, I did all that and triple-checked it. – sveti petar Jun 29 '22 at 09:05
  • @MartinZeitler That GET returns a 404. – sveti petar Jun 29 '22 at 10:20
  • 1
    I suggest creating a second service account and repeating the process. It's not happened for a few years now, but I've twice had unexplained instances similar to this where deleting the device account was the only solution. – Robbie Jun 29 '22 at 13:25
  • I will try when I start work tomorrow. Also, do you know if it's possible that some other gmail account also being an owner here could have any relevance? – sveti petar Jun 29 '22 at 13:52
  • @svetipetar Assuming you've passed the correct `url-encoded_url` string, there might be no meta-data for that domain-name. Without authentication I at least get a `401 UNAUTHENTICATED`. Wouldn't be certain the `google_auth_config.json` is readable or valid, because this is what could potentially cause a `403 FORBIDDEN`. Check it's file-system permissions - or use `xdbug` and set a break-point. – Martin Zeitler Jun 29 '22 at 15:41
  • RE "Also, do you know if it's possible that some other gmail account also being an owner here could have any relevance?" Highly unlikely. We have some projects with two service accounts; if we accidentally overstep some API limits in dev we still have our other service account for production. Don't know for indexing, but I presume their permission engine is similar. – Robbie Jul 01 '22 at 01:21

3 Answers3

0

Fist thing to do is to figure out which Forbiden error you are getting. I would like for you to try to get the full error message back.

forbiden - Indexing API has not been used in project

I just ran this. Using a service account the full error i got back was

The service indexing has thrown an exception.
HttpStatusCode is Forbidden.
Google.Apis.Requests.RequestError
Indexing API has not been used in project 1015451674269 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/indexing.googleapis.com/overview?project=1015451674269 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. [403]
Errors [
    Message[Indexing API has not been used in project 101574269 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/indexing.googleapis.com/overview?project=101574269 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.] Location[ - ] Reason[accessNotConfigured] Domain[usageLimits]
]

This error means that you have not enabled the indexing api under library in google cloud console. Have you done that.

Forbidden Failed to verify the URL ownership.

If i then make a request and ask for data on a url i have no access to I get.

The service indexing has thrown an exception.
HttpStatusCode is Forbidden.
Google.Apis.Requests.RequestError
Permission denied. Failed to verify the URL ownership. [403]
Errors [
    Message[Permission denied. Failed to verify the URL ownership.] Location[ - ] Reason[forbidden] Domain[global]
]

Which means the service account has no access to the url i am trying to access.

After a lot of digging I finally found the following link Grant owner status to your service account

By going to [Webmasters Verification}(https://www.google.com/webmasters/verification) I was able to add the service account and grant it access via delegation to my account.

enter image description here

Linda Lawton - DaImTo
  • 106,405
  • 32
  • 180
  • 449
  • Thanks, but what user? I don't understand Google's logic. I copied the code directly from Google's documentation for the PHP library. I created this from my Google account, so if I use that same Google account to access the API, what additional user needs to "authorize my request"? – sveti petar Jun 27 '22 at 10:55
  • "The Indexing API allows any site owner to directly notify Google when pages are added or removed. This allows Google to schedule pages for a fresh crawl, which can lead to higher quality user traffic." The owner of the website your trying to access indexing information for. This is private user data you need the permission of the owner of the data to access it. Thats why you use Oauth2. You may be able to grant access to the service account though the google search console. I need to try – Linda Lawton - DaImTo Jun 27 '22 at 10:58
  • That's the thing, I already gave the "owner" role to the service account which, according to the docs, is supposed to be all it takes in terms of permissions. – sveti petar Jun 27 '22 at 11:05
  • want to take a picture of that setting? blur out the service account name if you want. – Linda Lawton - DaImTo Jun 27 '22 at 11:09
  • Here it is: https://imgur.com/a/Neq0qwJ – sveti petar Jun 27 '22 at 11:17
  • Where is that from? Is this web page listed under a google workspace account or something? I have tried adding it in https://search.google.com/search-console/users? and it docent appear to grant the service account access. – Linda Lawton - DaImTo Jun 27 '22 at 11:46
  • It's https://console.cloud.google.com/iam-admin/iam?authuser=0&project=project-name-here&supportedpurview=project – sveti petar Jun 27 '22 at 12:00
  • Try https://www.google.com/webmasters/verification see update – Linda Lawton - DaImTo Jun 27 '22 at 12:32
  • I see the property there and the service account is listed as an owner there as well. It says "ownership was delegated by mygmailaccount@gmail.com" 5 days ago. – sveti petar Jun 28 '22 at 08:40
  • Could this be related to https/http or www in the URLs? I don't think so (used https://example.com everywhere) but worth a thought. – sveti petar Jun 29 '22 at 09:06
  • Without knowing the full error message you are getting its hard to track down the issue. – Linda Lawton - DaImTo Jul 02 '22 at 10:45
0

See Indexing API-specific errors:

Permission denied. Failed to verify the URL ownership.
User did not complete the Ownership Verification process
or is trying to update a URL that they do not own.

However, the provided response does not state this error message -
which suggests, that this is a general HTTP 403 and not a specific one.

In order to obtain more information about the cause, impersonate the service account with roles/iam.serviceAccountUser - or filter the Cloud Logs Explorer for the service account.

What if Google hasn't crawled that URL at all, so that it cannot update it?
I'm not certain, but I would try an URL which had been indexed already.
Try to get the notification status as described here: Get notification status.

Martin Zeitler
  • 1
  • 19
  • 155
  • 216
  • I have no idea what "impersonate the service account with roles/iam.serviceAccountUser" means. I don't normally use Google Cloud services so any step-by-step guide would be appreciated. The URL should be crawled, Google has had results from the site for months now. – sveti petar Jun 29 '22 at 09:08
  • `storage_path('google_auth_config.json')` might not be readable or valid (assuming that WMT and the service-account were setup properly); that "this works" comment alone is suspicious to me. And it's not my job to teach you Cloud IAM; better teach yourself how to debug PHP code with `xdebug`. – Martin Zeitler Jun 29 '22 at 15:52
  • "This works" meams I've var_dump-ed the content of that variable and it's correct. – sveti petar Jun 30 '22 at 06:21
  • It's not your job to do anything for me, really - you're here offering voluntary help and if that doesn't include Cloud services, no worries. – sveti petar Jun 30 '22 at 06:26
-1

HTTP 403 is an HTTP status code meaning access to the requested resource is forbidden. The server understood the request, but will not fulfill it.

This is the definition of error 403, so you have to check some corners!

First of all you might check your Private and Public keys are correct and belong to your desired domain(s).

If all above were ok, check your host (or server) is not banned in GOOGLE DNS servers. You can use IP tracker / tracers do find it.

At the end, check if IPs from your country are not banned! If so, you have to use some special protocols like L2TP!

MojSkin
  • 81
  • 1
  • 5
  • When you don't know anything about sanctions and no site (or service) never blocked your request, you have to give me a MINUS... Sorry if my problem and solution was'nt comprehensible for you! I just used a VPN connection to solve my connection to IMDB api... – MojSkin Jul 05 '22 at 11:36