7

A few days ago several thousand errors started pouring in to Google Stackdriver about Error: too many index entries for entity from one of our Cloud Functions.

Our code hasn't changed in a while so I believe the error is due to something in Firestore changing and causing queries or write operations to fail in this function.

The full stacktrace is below.

Note that none of the files referenced are code that we wrote, only Google's internal stuff for functions.

Error: too many index entries for entity
    at Http2CallStream.call.on (/srv/functions/node_modules/@grpc/grpc-js/build/src/client.js:96:45)
    at Http2CallStream.emit (events.js:194:15)
    at Http2CallStream.EventEmitter.emit (domain.js:459:23)
    at process.nextTick (/srv/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

Brief research made us believe that we had too many documents that were covered under an index. The collection being accessed by the function has around 35,000 documents right now. We deleted the one composite index we had created but the error continued to occur. The error will cause the function to crash and the HTTP request that triggered it will return an error page. This is not consistently reproducible and seems to be happening at random (around 5% of requests are failing).

I am happy to provide more information about our Firestore setup or our function source code if necessary. Any help is greatly appreciated.

Jonah Snider
  • 324
  • 1
  • 4
  • 11
  • 1
    Have you tried deleting the database? – OverCoder Dec 27 '19 at 10:47
  • 2
    You cannot have more than 20,000 fields in one document. Check this question: https://stackoverflow.com/questions/56428608/maximum-number-of-fields-for-a-firestore-document/61287286#61287286 – Andre Pena Apr 18 '20 at 10:01

1 Answers1

6

According to this documentation single-field indexes "stores a sorted mapping of all the documents in a collection that contain a specific field". If your database has 35000 documents it hit limits that you may find in the same document. This might be index entries per document or in some map or array.

I suppose that you have to setup some "Single-field index exemptions" to avoid this problem. All links needed are in the doc.

vitooh
  • 4,132
  • 1
  • 5
  • 16
  • 1
    Looks like this was the issue. Several of our documents have giant arrays of hundreds of items which were probably making Firestore angry. We just told it to stop indexing that field and will check to see if it improves. – Jonah Snider Dec 27 '19 at 20:05
  • It still failed, even with 30k. Set the maxlen now to 15k before sampling down my array in a document that's failing to be written. – Boern Jul 04 '22 at 18:39