1

I am investigating Cygnus for FIWARE Orion historical data persistence.

Since Cygnus 3.0.0, indexes are created according to the data model when writing to MongoDB, but the order of the indexes created and the data written to MongoDB are different.

I tried it and found that the following indexes were created.

> db['sth_/_Car1_Car'].getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_"
        },
        {
                "v" : 2,
                "key" : {
                        "recvTime" : 1,
                        "attrName" : 1,
                        "attrType" : 1,
                        "attrValue" : 1
                },
                "name" : "cyg_raw_opt"
        }
]

However, the data written from Cygnus to MongoDB is in the following order, so it seems that the index is not valid.

> db['sth_/_Car1_Car'].find()
{ "_id" : ObjectId("6475a09a91bf560e1ef4cb0d"), "attrName" : "speed", "attrType" : "Float", "attrValue" : 80, "recvTime" : ISODate("2023-05-30T07:07:05.588Z") }
{ "_id" : ObjectId("6475a09f91bf560e1ef4cb0e"), "attrName" : "speed", "attrType" : "Float", "attrValue" : 70, "recvTime" : ISODate("2023-05-30T07:07:11.584Z") }

I checked some past Issues and found that the data was lined up in the same order as the index.(ex. https://github.com/telefonicaid/fiware-cygnus/issues/2204)

What settings are needed to make this order?

My environment is as follows:

  • Cygnus 3.1.0
  • MongoDB 4.4

Thank you in advance.

fgalan
  • 11,732
  • 9
  • 46
  • 89
ctc-watanabe
  • 31
  • 4

1 Answers1

1

MongoDB normally stores the documents (in BSON format in the storage backend) with the fields in the same order they appear in the insert() operation (done by the MongoDB Java driver, in the case of Cygnus). However, order doesn't matter for the most of MongoDB functionality and index usage is not an exception.

An index is basically a data structure that "points" to documents in the database. So, no matter if recvTime comes before attrName in the document stored in the storage backend, as the "document address" for that document, pointed by the index, doesn't change.

fgalan
  • 11,732
  • 9
  • 46
  • 89
  • Thanks for your reply and sorry for the delay on my response. You are correct, I checked with ".explain()" and the index was working. Sorry, I didn't explain enough about the situation where I felt the query was slowing down. When I use STH-Comet (2.10.0) to get historical data, the response time has increased. Writes by Cygnus 3.1.0 were acquired with STH-Comet in about 0.1-0.2 seconds, whereas writes by Cygnus 2.20.0 were acquired in about 0.01-0.02. Is it possible that STH-Comet is not compatible with Cygnus upgrades? – ctc-watanabe Jun 23 '23 at 08:02
  • 1
    There isn't any known slowdown on STH performance due to Cygnus upgrades. Not sure what do you mean by "writes by Cygnus were acquired with STH-Comet"... The write operations done by Cygnus are independent of the read operations done by STH-Comet. – fgalan Jun 27 '23 at 14:40
  • I am sorry I did not explain myself better. I have created the following post with details of my running environment and logs, please check here. https://stackoverflow.com/questions/76570277 Regarding this post (about Cygnus and MongoDB indexing), I understood that there is no problem. – ctc-watanabe Jun 28 '23 at 05:38