mongoid: Rails says field exists, mongodb says contrary - ruby-on-rails

Question

I've got a collection of complex documents.
{
<some fields>,
meta_info: {
company_name: String,
invalid: Boolean,
mobile_layout: String,
<more fields>
}
<lots more fields>
}
I ask Rails to find me all those documents where meta_info.invalid is true/false/nil using
finished_acts.where('meta_info.invalid' => true/false/nil)
Now there is ONE document where the field does not exist. I ask...
finished_acts.find('meta_info.invalid' => {'$exists' => false})
=> nil
which is simply untrue (plus it also yields nil if I ask {'$exists' => true}), and so is
finished_acts.where('meta_info.invalid' => {'$exists' => false}).count
=> 0
How can I find this document? I've spent days with a collection of statistical data which was always off by one count when compared to the info given me by the database itself, and this nonexistent field was the reason.
I am using mongoDB V3.4.17 and Mongoid 6.1.0.
EDIT: I've since learned that I used the .find command incorrectly, it is only intended to be used for the _id field and does not accept JSON/Hashes.
My problem obviously shows a bug in the implementation of the Active Record adaptation of Mongoid, and I am slowly converting my code to always use aggregations. When doing so, I get the correct number of documents. Of course, the structure returned by aggregations is more complex to handle since it is only hashes and arrays, but if that's the trade-off for getting correct results I am happy with it.

Answer 1

Don't rely on the ActiveRecord adaptation in Mongoid; use the aggregation pipeline. This will (to the best of my knowledge) always return correct results since it simply pushes the argument hash into mongo.
The aggregation pipeline at first seems unintuitive to use, but after you get to know how to use it it is a very powerful tool if you want to make complex queries. The Rails syntax is as follows:
MyClassName.collection.aggregate( <array> )
where the <array> contains hashes, each of which is a command used on the result of the execution of the preceding hash. The documentation can be found at https://docs.mongodb.com/manual/aggregation/.
To convert the commands there to Ruby it is only required to surround the entries by quotes.
To give an example: The following is the mongo syntax:
db.orders.aggregate([
{ $match: { status: "A" } },
{ $group: { _id: "$cust_id", total: { $sum: "$amount" } } }
])
This takes all documents from the orders collection and selects all those where the status field matches "A" (the word $match left of the colon is a mongo command). Those then get grouped ($group) by the cust_id field and the sum ($sum) of the contents of the amount field is computed.
If you want to convert this to Ruby, you change it into
Orders.collection.aggregate([
{ '$match': {'status': 'A'}},
{ '$group': {'_id': '$cust_id', 'total': {'$sum': '$amount'}}}
])
This worked for me, and what's even better is that it takes significantly less time than using Orders.where(...) in Rails and doing the computation in Rails.
The trade-off is that you don't get returned Ruby objects but hashes of arrays of hashes.