mongoid: Rails says field exists, mongodb says contrary

Question

I've got a collection of complex documents.

{
<some fields>,
meta_info: {
   company_name: String,
   invalid: Boolean,
   mobile_layout: String,
   <more fields>
  }
<lots more fields>
}

I ask Rails to find me all those documents where meta_info.invalid is true/false/nil using

finished_acts.where('meta_info.invalid' => true/false/nil)

Now there is ONE document where the field does not exist. I ask...

finished_acts.find('meta_info.invalid' => {'$exists' => false})
 => nil

which is simply untrue (plus it also yields nil if I ask {'$exists' => true}), and so is

finished_acts.where('meta_info.invalid' => {'$exists' => false}).count
 => 0

How can I find this document? I've spent days with a collection of statistical data which was always off by one count when compared to the info given me by the database itself, and this nonexistent field was the reason.

I am using mongoDB V3.4.17 and Mongoid 6.1.0.

EDIT: I've since learned that I used the .find command incorrectly, it is only intended to be used for the _id field and does not accept JSON/Hashes.

My problem obviously shows a bug in the implementation of the Active Record adaptation of Mongoid, and I am slowly converting my code to always use aggregations. When doing so, I get the correct number of documents. Of course, the structure returned by aggregations is more complex to handle since it is only hashes and arrays, but if that's the trade-off for getting correct results I am happy with it.

`$exists: false` works on nested fields: https://mongoplayground.net/p/be8sVcdQU8g. ruby/mongoid _shouldn't_ mess this up, but try querying using mongodb cli anyway — Sergio Tulentsev, Jul 09 '21 at 08:54
Note that missing field will pass the null check: https://mongoplayground.net/p/P2dhpqwNvMN https://docs.mongodb.com/manual/tutorial/query-for-null-fields/#equality-filter — Sergio Tulentsev, Jul 09 '21 at 08:56
BTW, I know I can do it with an aggregation, but this yields a collection of documents which are awkward to manipulate. I'd like to use the models Mongoid provides. — MDickten, Jul 09 '21 at 09:05
@SergioTulentsev Sorry, I know how to do it with mongo. I want to know how to do it with Mongoid. I have a Rails program and it needs to find the correct result set. — MDickten, Jul 09 '21 at 09:06
Well, let's look at the query mongoid generates, then. I'm curious, what would be the difference. — Sergio Tulentsev, Jul 09 '21 at 11:12
This is another problem. I can look at the server output, but all the generated mongo statements are truncated. There's a SO thread on it (https://stackoverflow.com/questions/17960230/how-can-i-see-raw-mongodb-queries-with-mongoid), but nobody knows a real answer. If you know how to see the generated statements, please write an answer, I'd be much obliged! — MDickten, Jul 09 '21 at 14:45
Can you try it locally? Shouldn't be truncated in dev env, I think. — Sergio Tulentsev, Jul 09 '21 at 15:04
It is, unfortunately. And it seems that it's mongo (and not the Rails server) that does the truncating. I've started messing around with the mongo verbosity settings in the .conf file, and if I get anywhere, I'll answer that old thread. — MDickten, Jul 09 '21 at 17:08

score 0 · Accepted Answer · answered Aug 04 '21 at 08:36

Don't rely on the ActiveRecord adaptation in Mongoid; use the aggregation pipeline. This will (to the best of my knowledge) always return correct results since it simply pushes the argument hash into mongo.

The aggregation pipeline at first seems unintuitive to use, but after you get to know how to use it it is a very powerful tool if you want to make complex queries. The Rails syntax is as follows:

MyClassName.collection.aggregate( <array> )

where the <array> contains hashes, each of which is a command used on the result of the execution of the preceding hash. The documentation can be found at https://docs.mongodb.com/manual/aggregation/. To convert the commands there to Ruby it is only required to surround the entries by quotes.

To give an example: The following is the mongo syntax:

db.orders.aggregate([
  { $match: { status: "A" } },
  { $group: { _id: "$cust_id", total: { $sum: "$amount" } } }
])

This takes all documents from the orders collection and selects all those where the status field matches "A" (the word $match left of the colon is a mongo command). Those then get grouped ($group) by the cust_id field and the sum ($sum) of the contents of the amount field is computed. If you want to convert this to Ruby, you change it into

Orders.collection.aggregate([
      { '$match': {'status': 'A'}},
      { '$group': {'_id': '$cust_id', 'total': {'$sum': '$amount'}}}
   ])

This worked for me, and what's even better is that it takes significantly less time than using Orders.where(...) in Rails and doing the computation in Rails. The trade-off is that you don't get returned Ruby objects but hashes of arrays of hashes.

mongoid: Rails says field exists, mongodb says contrary

1 Answers1