7

I have a small project involving some simple financial time-series data with some real-time components on the front end. I was hoping to use the Firebase infrastructure since it offers a lot of things without having to set up much infrastructure, but upon investigating it doesn't seem to be a good choice for storing time series data.

Admittedly, I have more experience with relational databases so it's possible I am asking an extremely basic question. If I were to use Firestore to store time-series data, could someone provide an example of how one might structure it for efficient querying?

Am I better served using something like Postgres?

blueether
  • 3,666
  • 4
  • 26
  • 32
  • 2
    If you need efficient queries for time series you'd rather go for a time series database (tsdb) imo. My personal preference go for Warp10 since you can do the most complex queries with it by far. – user2682877 Aug 07 '19 at 09:49
  • In order for us to present a solution, we would need to understand what you are querying for and what the expected end result is. Also, asking an opinion type question is off topic - we wouldn't be able to answer that anyway because we don't know the use case. – Jay Aug 07 '19 at 18:34
  • 2
    @Jay The data is described as 'simple financial time-series data' in the fist line. The usecase is described as 'real-time components on the frontend.' With time-series data, querying by datetime ranges is generally the desirable. If the question needs clarification, then I could ask - Is Firestore appropriate for indexing and querying by datetime for real-time charting? The second part of the question is certainly asking for an educated opinion, but the first part should be pretty clear – blueether Aug 07 '19 at 20:29
  • It's vague as pretty much any database can query timestamps (aka 'data'). Is there a specific example you can present that describes your data and maybe an example of a query you want to run? It's a good idea to include code and structures in your question as well so we can see what you've tried and what isn't working. Please take a moment and review [How do I ask a good question?](https://stackoverflow.com/help/how-to-ask) and the very important [How to create a Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve) – Jay Aug 08 '19 at 12:51
  • Oh, and the Firestore Getting Started Guide [Queries](https://firebase.google.com/docs/firestore/query-data/queries) provides some great examples of simple and compound queries including limits and ranges. Once you work through that code, it may provide a better feel for Firestore capabilities. – Jay Aug 08 '19 at 12:54
  • @Jay Some of the others that have answered understand the inherent sequential and increasing nature of time and have thus offered suggestions involving time-series databases. Contrary to what you believe, timestamps have exploitable structure and are not just data. This allows more efficient treatment for fast querying if the database is optimized for this very common data pattern. Based on what I can gather from other answers, Firestore is a Document Store and thus appears poorly equipped to select only datetime ranges I want, rather it filters out ones I don't. – blueether Aug 08 '19 at 14:59
  • This article specifically about [Time Series Databases](https://web.archive.org/web/20190626143018/https://www.techrepublic.com/article/why-time-series-databases-are-exploding-in-popularity/) provides info on this topic. Note: *Relational and NoSQL databases can be used for time series data, but arguably developers will get better performance from purpose-built time series databases* Firebase is a NoSQL database. If you are looking for a Time Series Database there are many options including NoSQL - it depends on the use case, which was not included in the question, which is why its closed. – Jay Aug 11 '19 at 13:33
  • Note that Warp timestamps are signed longs, which is the same context in which I was using the word 'timestamp'. If you feel that *timestamps have exploitable structure*, then perhaps another storage method would be in order as [Postgres also stores timestamps](https://www.postgresql.org/docs/current/datatype-datetime.html) – Jay Aug 11 '19 at 13:45

1 Answers1

2

Probably best bet would be to use a time-series database. Looks like Warp 10 has already been mentioned (https://www.warp10.io).

The benefit of something like warp is the ability to query on the time component of your database. I believe firebase only has simple greater/lesser than queries available for time.

Jan-Michael Tressler
  • 629
  • 5
  • 17
  • Thanks for the link, that was very helpful. I'll probably go with this or something more relational like Postgres rather than a Document Store. I also found this link helpful: https://stackoverflow.com/questions/4814167/storing-time-series-data-relational-or-non – blueether Aug 08 '19 at 15:08
  • To clarify, Firestore doesn't have a *time* component, it can store numbers (double). Keeping in mind that warp10 timestamp [ts](https://www.warp10.io/content/02_Getting_started#data-format) **Timestamp is a signed LONG**. Firestore has queries that are greater/lesser/equal and ranges as well, which are similar to warp queries, and can query on a 'time' component. As you can see from [Fetch Docs](https://www.warp10.io/content/04_Tutorials/01_WarpScript/20_Master_GTS#basic-fetch), queries are similar. For future readers, can you clarify this answer as it's unclear what the answer actually is? – Jay Aug 11 '19 at 13:22
  • 1
    @Jay, Warp10 has the ability to perform analytical time-series queries and calculations server side, whereas Firestorm requires a query, followed by these analytical calculations to be done client side. While someone could use Firestorm + Google Cloud Functions to approximate the Warp10 analytics, it might be too low-level for some users. Here's a good briefing on Warp10's capabilities https://medium.com/@PierreZ/engage-maximum-warp-speed-in-time-series-analysis-with-warpscript-c97a9f4a0016 – Jan-Michael Tressler Aug 11 '19 at 19:10