How to send multipart/mixed request for google indexing batch request in NodeJs?

Question

I am using Nodejs to connect with GoogleApis v35.0.0 to tell Google to update or remove pages from the Google index. And I stuck in the multipart/mixed request, the body of multipart when I send the request through Google indexing batch request.

I could able to send an individual page update request to Google by following the indexing API documentation. But since Google has the limited quota at maximum of 200 requests per day and I need to update more URL's than that. So, I am trying to use google indexing batch request which can group at maximum of 100 individual requests and it counts as 1 request.

I am having issue with the correct format of multipart body when I am trying to send the request by batch. I am using JWT (JSON Web Token) of GoogleApis which extended from oauth2 to authenticate my account and using the request library v2.88.0 to send the request to Google.

Since request library already handle the multipart boundary that's why I am not sending that as one of the request options information. I also check the information in the multipart/mixed of the request npm library but I only found the one similar but not the same which is the multipart/related (https://github.com/request/request#multipartrelated).

According to the piece of batch request body example from Google, I need to use multipart/mixed as content type in the main request:

POST /batch HTTP/1.1
Host: indexing.googleapis.com
Content-Length: content_length
Content-Type: multipart/mixed; boundary="===============7330845974216740156=="
Authorization: Bearer oauth2_token

--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+2>

POST /v3/urlNotifications:publish [1]
Content-Type: application/json
accept: application/json
content-length: 58

{ "url": "http://example.com/jobs/42", "type": "URL_UPDATED" }

Here's my code:

    return jwtClient.authorize(function(err, tokens) {
      if (err) {
        console.log(err);
        return;
      }

      let options = {
        url: 'https://indexing.googleapis.com/batch',
        method: 'POST',
        headers: {
          'Content-Type': 'multipart/mixed'
        },
        auth: { 'bearer': tokens.access_token },
        multipart: [
          {
            body: JSON.stringify({
              headers: {
                'Content-Type': 'application/http'
              },
              method: 'POST',
              url: 'https://indexing.googleapis.com/v3/urlNotifications:publish',
              body: {
                'Content-Type': 'application/json',
                url: 'https://www.test.com/es/1234',
                type: 'URL_UPDATED'
              }
            })
          }
        ]
      };

      request(options, function (error, response, body) {
        console.log(body);
      });

    });

I am getting error in the body of multipart, I don't know which kind of body google indexing batch request is waiting for. Seems like everything inside the body of multipart are considering as headers. But according to the documentation the format of batch request, it says that "Each part begins with its own Content-Type: application/http HTTP header. The body of each part is itself a complete HTTP request, with its own verb, URL, headers, and body". For more details information check: https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch.

However, I am getting the following error when I execute my code:

{
  "error": {
    "code": 400,
    "message": "Failed to parse batch request, error: Failed in parsing HTTP headers: {\"Content-Type\":\"application/http\",\"method\":\"POST\",\"url\":\"https://indexing.googleapis.com/v3/urlNotifications:publish\",\"body\":{\"Content-Type\":\"application/json\",\"url\":\"https://www.test.com/es/1234\",\"type\":\"URL_UPDATED\"}}\n. Received batch body: ",
    "status": "INVALID_ARGUMENT"
  }
}

Does someone know which is the correct format of body inside the multipart when it request to google indexing batch request?

Thanks in the advance!

Answer 1

Batching does not help avoid quota limits

I could able to send an individual page update request to Google by following the indexing API documentation. But since Google has the limited quota at maximum of 200 requests per day and I need to update more URL's than that. So, I am trying to use google indexing batch request which can group at maximum of 100 individual requests and it counts as 1 request.

There is nothing in batching that states it only counts as one against your quota.

While batching can save you on the overhead of constructing many HTTP requests each Google APIs request within a batch request will count against your daily project quota. By default a project can make up to 200 request per day; batching will not help you stay below this quota.

Apply for a higher quota

Have you considered applying for a higher quota? I know it can take time to get the response back, but you may just want to wait and see what they say.

Note google-apis-nodejs-client

The library does not support batching so you are going to have to do it yourself as you are currently #1130

Your Actual issue

Let me know if you want to continue trying to get batching working. I will see if i can help. With manual version.

Answer 2

As @DalmTo says the quota will still apply, even to batch requests. But also you are not correctly constructing the payload, the following example works.

const items = batch
  .filter(x => x)
  .map(line => {
    return {
      'Content-Type': 'application/http',
      'Content-ID': batchId,
      body:
        'POST /v3/urlNotifications:publish HTTP/1.1\n' +
        'Content-Type: application/json\n\n' +
        JSON.stringify({
          url: line,
          type: 'URL_UPDATED',
        }),
    };
  });
const options = {
  url: 'https://indexing.googleapis.com/batch',
  method: 'POST',
  headers: {
    'Content-Type': 'multipart/mixed',
  },
  auth: { bearer: access_token },
  multipart: items,
};
request(options, (err, resp, body) => {
  //...
});