0

I am populating a series of dates and temperatures that I was thinking of storing in a Firestore Database to later be consumed by the front-end with the following structure:

{
   date: ['1920-01-01', '1920-01-02', '1920-01-03', '1920-01-04', '1920-01-05', ...],
   values: [20, 18, 19.5, 20.5, ...]
}

The array may consider a lot of years, so it turns huge, with thousands of entries. Firestore database started complaining about returning the too many index entries for entity error, and even if I get the data uploaded, the user interface Firebase -> Firebase Database -> Panel View collapses. That happens even with less than 3000 entries array.

The fact is that the data is consumed in the front-end with an array structure very similar to the one described above (I want to plot it using Echarts library). This way, I found this structure to be the more natural way, as any other alternative will require reversing the structure to arrays in the front-end.

Nevertheless, I see that Firestore Database very clearly does not like this structure. What should I do? What is the best practice for dealing with this kind of data in Firestore?

Alex Mamo
  • 130,605
  • 17
  • 163
  • 193
Rodrigo_V
  • 130
  • 7

2 Answers2

1

The indexes required for the most basic queries in Firestore are automatically created for you. However, there are some limits involved. So you're getting the following error:

too many index entries for entity

Because you hit the maximum number of index entries for a document, which is 40,000. If you add too many elements into an array or you add too many fields to a document, then you can reach the maximum limit.

So most likely the number of elements that exist in the date array + the number of elements that exist in the values array is bigger than 40k, hence the error.

To solve this, you might consider creating two separate documents, one for each array. If you still hit the maximum limit, then you might consider creating a document for each hour, and not for an entire day. In this way, you'll drastically reduce the number of elements that exist in an array.

If you don't find these solutions useful, then you have to set some "Single-field index exemptions" to avoid the above error.

Alex Mamo
  • 130,605
  • 17
  • 163
  • 193
  • Thanks for your answer. Do you know if there is any way to set the date field as the index? In this case I could create separate documents for the different field arrays. For example, avg_temperature, max_temperature, rainfall... all of them indexed by date. Can I define the index with the .set() python method? – Rodrigo_V Nov 17 '22 at 17:38
  • As I said before, the indexes for the most basic queries are automatically created. But those indexes are created on the server, and not by users. The set() function cannot create an index. If you need complex indexes then you have to [create them manually](https://stackoverflow.com/questions/50305328/firestore-whereequalto-orderby-and-limit1-not-working) in the Firebase console. – Alex Mamo Nov 17 '22 at 17:50
  • Hey Rodrigo. Did my answer help? Can I help you with other information? – Alex Mamo Dec 02 '22 at 12:21
1

Firestore is not the best tool to deal with time series. The best solution I found in Firestore was creating an independent document for each day in my data. Nevertheless, that raises the number of documents I need to fetch from the front-end side and, therefore, the costs. By using large arrays in Firestore, you easily reach the index limit, and you are forced to remove the index, which I feel is a big red flag, suggesting checking another tool.

The solution I found, in case is useful for anyone, was building my API in Flask using MongoDB as a database. Although it takes more effort than just using Firestore, it deals better with time series and brings more flexibility.

Rodrigo_V
  • 130
  • 7