0

I need to store high frequency, periodic time-series data of some metrics in mongodb (after every 5 seconds), since the normal approach is too expensive:

   {
      timestamp: ISODate("2013-10-10T23:06:37.000Z"),
      type: String,
      value: Number
    },

I want to write an optimized one as mentioned here, the document would look like:

{
  timestamp_hour: ISODate("2013-10-10T23:06:00.000Z"),
  type: “memory_used”,
  values: {
    0: 999999,
    …
    37: 1000000,
    38: 1500000,
    … 
    720: 2000000,
  }
}

in the above model instead of storing a document every 5 seconds,I am storing a document every 1 hour and storing all the data generated in that particular hour in the "values" object as key value pair.

To keep the document size as minimum as possible and to increase the performance while fetching the data, should I use Array to store the values or dynamic object, which one will give me the smallest size and fastest read( I will be needing chunks of values to read in any specific document)?

and if I use the object, How to define the schema for it since the number of values( 720 in above example) is not constant it can be changed.

Neil Lunn
  • 148,042
  • 36
  • 346
  • 317
Bilal Alam
  • 874
  • 10
  • 26
  • There are 720 5-seconds intervals in an hour. It's been like that for a long time already and is not going to change anytime soon. It should be safe to assume it is constant, at least in context of a software application lifetime. – Alex Blex May 24 '18 at 21:13
  • 1
    You probably just mean `Mixed` here or even turn of the schema verification altogether with `{ strict: false }` and don't specify the key in schema at all. – Neil Lunn May 24 '18 at 22:07
  • @AlexBlex what I actually meant was the 5-seconds Interval is not constant instead so we can have less or more than 720 values in the object. – Bilal Alam May 25 '18 at 09:56
  • @BilalAlam, do you mean it is not always 5 seconds, i.e. a random interval each time, or it is not each 5-seconds interval is recorded, i.e. some intervals are missing? Either way, how you imagine it would work with arrays where you have no control on keys? – Alex Blex May 25 '18 at 10:06
  • @AlexBlex Actually the application pings a list of IP address after every 5 seconds(Default value) and store the latency of each ping, this interval time can be changed by user.So basically the interval time wont be different for each ping but it may change at any given time. – Bilal Alam May 25 '18 at 10:23
  • 1
    If it can be changed by user at any time, it is random (not cryptographically random, but rather unpredictable random). Array is not gonna work then. Keys in array are strictly incremental. – Alex Blex May 25 '18 at 10:36
  • You are right, there is now way I can use Arrays and track pings sent within an hour after the interval time is changed by user. – Bilal Alam May 25 '18 at 10:50

0 Answers0