Administration access using the Firestore REST API

Question

We've found that in our use-case the Firestore REST API is a lot more performant than using the firebase-admin's default database connection for reading data.

We're using Firestore on our server, and there's some sensitive data we have stored on lock-down by using the read/write: false rules. This way only the server can read this data from our service account.

Because of performance reasons regarding the Firestore handshake process, it's much faster for us (as we're using lambda) to use the REST API to request data from the database.

However, we've having problems accessing the data that's on lockdown, because it seems the only way to authorize the REST API is with a client's authorization token.

We need a way to have administrative database access via the REST API, is there some special authorization header or API key that we can't find somewhere that will allow this?

I'm kind of surprised that the SDK is slower than REST. Do you have benchmarks to share? — Doug Stevenson, Aug 05 '18 at 16:16
We have several, the initial get request on firestore takes 1.5-2 seconds, requests that follow this are undeniably faster, but since we issue on average 1-2 requests per endpoint, the 400-600ms of the REST API is far more performant. This is talked about in several links [Link1](https://stackoverflow.com/questions/46717898/firestore-slow-performance-issue-on-getting-data) [Link2](https://stackoverflow.com/questions/46675765/firestore-document-get-performance) [Link3](https://groups.google.com/forum/#!topic/firebase-talk/MpS6C-ZxdZ0) — Christian Tucker, Aug 05 '18 at 19:24
Also, https://stackoverflow.com/questions/40165122/why-does-the-first-firebase-call-from-the-server-take-much-longer-to-return-than — Christian Tucker, Aug 05 '18 at 19:25
@DougStevenson Here's some links, although they're not our direct benchmarks, I can provide them if you need :) — Christian Tucker, Aug 05 '18 at 19:26
It's good to note that since we're using lambda, almost every request to our server issues a new Firestore handshake / "cold start" / whatever you want to call it. In that last link I put, one of the comments explains this in detail, and how he used firebase-queue with an EC2 instance to get around it. We opted for the REST API as we don't want to manage an EC2 instance. — Christian Tucker, Aug 05 '18 at 19:28
First access is always the slowest because you're paying the cost of the lazy initial connection. It's better if you can arrange thing to reuse its persistent connection. — Doug Stevenson, Aug 05 '18 at 19:29
@DougStevenson We've looked into that, but our containers switch between different firebase instances and aws never assures that a container will be re-used. Thus it's impossible to rely on a persistent connection. When we attempted caching connections we ran into the issue of data ending up in wrong databases as-well, from a connection being opened from the last request that accessed a different database from the current one. Obviously this could be prevented with rules, but the bigger issue here was reusing the connection, so regardless we had to manually close it after each lambda call. — Christian Tucker, Aug 06 '18 at 04:05

score 0 · Accepted Answer · answered Aug 05 '18 at 16:15

0

You'll need to authenticate with the credentials of a service account that has permission to access the database. According to the documentation:

If you authenticate your requests with a service account and a Google Identity OAuth 2.0 token, Cloud Firestore assumes that your requests act on behalf of your application instead of an individual user. Cloud Firestore allows these requests to ignore your security rules. Instead, Cloud Firestore uses Cloud Identity and Access Management (IAM) to determine if a request is authorized.

The default App Engine service account for your project should already be able to do this. Once you have a token for your service account, you can pass it in the Authorization header of your requests, which is also mentioned in the docs.

answered Aug 05 '18 at 16:15

Doug Stevenson

297,357
32
422
441

Thanks for pointing this out, we use the `firebase-admin` library for NodeJS. So when we create our firebase instance, we do so with the JSON file using `firebase.credential.cert(...)`. I don't really understand what we're supposed to be passing into the API request or how we're supposed to get it from the documentation. – Christian Tucker Aug 05 '18 at 19:30
Did you follow the links in the documentation about how to get an oauth token for your service account? – Doug Stevenson Aug 05 '18 at 19:32
Yes, but I may be doing something wrong when creating the JWT token, here's what I have.. const jwt = require('jsonwebtoken') return jwt.sign( { algorithm: 'RS256', expiresIn: 3600, notBefore: Math.round(Date.now() / 1000), audience: 'https://firestore.googleapis.com/google.firestore.v1beta1.Firestore', issuer: this.serviceEmail, subject: this.serviceEmail, keyid: this.keyId }, this.privateKey ) – Christian Tucker Aug 06 '18 at 17:22
We're constantly getting Status code 401 after trying with oAuth – Christian Tucker Aug 06 '18 at 17:23
This sounds like a different question to ask - it's not really feasible to work this out in comments. – Doug Stevenson Aug 06 '18 at 18:18

Chris Chiasson · Answer 2 · 2023-08-24T01:29:57.670

Here is an example for node.js for reference.

import { GoogleAuth } from 'google-auth-library';


async function getCollectionAsync(collection) {
  try {
    // needs GOOGLE_APPLICATION_CREDENTIALS or equivalent to be set
    const googleAuth = new GoogleAuth({
      scopes: ['https://www.googleapis.com/auth/cloud-platform']
    });
    const googleAuthClient = await googleAuth.getClient();
    const tokenObj = await googleAuthClient.getAccessToken();
    const token = tokenObj.token;
    const projectId = googleAuthClient.projectId;
    const parent = `projects/${projectId}/databases/(default)/documents`;
    const url = `https://firestore.googleapis.com/v1/${parent}:runQuery`;
    const Authorization = `Bearer ${token}`;
    const headers = { Authorization, "Content-Type": "application/json" };
    const body = {
      structuredQuery: {
        from: [{ collectionId: collection }],
        limit: 40000, // OPTIONALLY: limit the number of documents

        // OPTIONALLY: limit which fields are returned for speed
        select: { fields: [{ "fieldPath": "__name__" }] }
      }
    };
    const response = await fetch(url,
      { method: "POST", headers, body: JSON.stringify(body) });
    return response.json();
  } catch (error) {
    console.error(`Error getting collection ${collection}`, String(error));
    throw error;
  }
}

Administration access using the Firestore REST API

2 Answers2