1

I have a MongoDB collection named market_data. It has documents which looks like this:

{

        "_id" : ObjectId("1235"),

        "nm" : " eod_ny",

        "asOfDt" : "2018-11-20",

        "rgnId" : 1,

        "mrktCntxtDfId" : ObjectId("32345")

        "rlsd" : false,

        "crtdOn" : ISODate("2018-11-21T02:05:12.331Z"),

        “observations” : [

{

    Key1: “abc”,

    Key2: : “def”

},

{

    Key1: “123”,

    Key2: : “456”

}



]

}

For most of the documents, the length of observation array is quite less, but for few records (around 20-30) daily the observation array has more than million records and it exceeds the 16 MB limit.

Is there any strategy I can apply to store this data without moving to GridFS.

I'm using MongoDB raw Java driver latest version in my application.

Neil Lunn
  • 148,042
  • 36
  • 346
  • 317
Mb-One
  • 41
  • 6
  • You cannot exceed the BSON Limit. That's why it's called a **limit**. The very simple course of action here is arrays with millions of records is simply not practical. Put the data in a separate collection instead. This has already been discussed and answered at length in long standing answers. – Neil Lunn Nov 21 '18 at 10:45
  • Can you point me out to an answer which has a solution to this, because I can't find any. Also I got to know a sequence approach, by adding a sequence key and breaking this object into child object. – Mb-One Nov 21 '18 at 10:54
  • There is no **one answer**, which is actually the point. You most likely "should" break up the array into it's own collection instead of embedding. But if you actually read the linked answers, then you should get a better understanding of why each concept is important. – Neil Lunn Nov 21 '18 at 11:02
  • Large (and potentially unbounded) arrays are a performance anti-pattern. You definitely need to reconsider your data model if it is approaching or exceeding 16MB per document. It sounds like your observations are likely to be time series data. There is a series of posts on the MongoDB blog which should help: [Time Series Data and MongoDB](https://www.mongodb.com/blog/post/time-series-data-and-mongodb-part-1-introduction). – Stennie Nov 22 '18 at 20:52

0 Answers0