How to escape $ while inserting inside mongoDB

Question

I have a large object which I have no control over, fields might vary a lot:

Error ==>> { MongoError: The dollar ($) prefixed field '$' in 'value.external.originalRequest.rawRequest.ns2:LoadData.Orders.0.Order.0.TotalThirdPartyCharge.0.CurrencyValue.0.$' is not valid for storage. 0|lt-check | at Function.MongoError.create (/home/ops/lt-checkin-api/node_modules/mongodb-core/lib/error.js:31:11) 0|lt-check | at /home/ops/lt-checkin-api/node_modules/mongodb-core/lib/connection/pool.js:497:72 0|lt-check | at authenticateStragglers (/home/ops/lt-checkin-api/node_modules/mongodb-core/lib/connection/pool.js:443:16) 0|lt-check | at Connection.messageHandler (/home/ops/lt-checkin-api/node_modules/mongodb-core/lib/connection/pool.js:477:5) 0|lt-check | at Socket. (/home/ops/lt-checkin-api/node_modules/mongodb-core/lib/connection/connection.js:333:22) 0|lt-check | at Socket.emit (events.js:159:13) 0|lt-check | at addChunk (_stream_readable.js:265:12) 0|lt-check | at readableAddChunk (_stream_readable.js:252:11) 0|lt-check | at Socket.Readable.push (_stream_readable.js:209:10) 0|lt-check | at TCP.onread (net.js:608:20) 0|lt-check | name: 'MongoError', 0|lt-check | message: 'The dollar ($) prefixed field \'$\' in \'value.external.originalRequest.rawRequest.ns2:LoadData.Orders.0.Order.0.TotalThirdPartyCharge.0.CurrencyValue.0.$\' is not valid for storage.', 0|lt-check | ok: 0, 0|lt-check |
errmsg: 'The dollar ($) prefixed field \'$\' in \'value.external.originalRequest.rawRequest.ns2:LoadData.Orders.0.Order.0.TotalThirdPartyCharge.0.CurrencyValue.0.$\' is not valid for storage.', 0|lt-check | code: 52, 0|lt-check |
codeName: 'DollarPrefixedFieldName' }

Here the key causing the error:

"rawRequest": {
                        "ns2:LoadData": {
                            "$": {
                                "xmlns:ns2": "http://schemas.3gtms.com/tms/v1/tns"
                            },
                            "BatchInfo": [
                                {
                                    "$": {
                                        "xmlns": ""
                                    },

As there are multiple keys starting with $.

Is there a setting in mongoose or mongoDB itself to insert anything without validating the request coming in.

I am not sure about the keys and moreover is there a good method to escape all the keys having $. @AnirudhSimha — Black Mamba, May 06 '19 at 06:11
check https://docs.mongodb.com/manual/reference/limits/ for reserved keywords.If it is a linux env its /\. "$ — Anirudh Simha, May 06 '19 at 06:14
Was just looking around and found this: https://www.npmjs.com/package/mongo-escape — Anirudh Simha, May 06 '19 at 06:18
No doubt, but how can you restrict $ from some third party API then all you can do is escape @Adam — Black Mamba, Nov 06 '19 at 10:08
@BlackMamba - do you need to be stored in mongo as a document like this? Maybe you just need to store it stringified and then just parse it whenever you need to? Maybe not, just curious. `$` in keys in mongo are for specific purposes, and, in this document, you aren't using them for that, which is what makes me believe you'd be completely fine with just storing the stringified version of it. — Adam Jenkins, Nov 06 '19 at 13:07
It was long time ago I've answered with what I used back then. But that resolved the issue so all good for me — Black Mamba, Nov 06 '19 at 14:16
@BlackMamba - somehow this thread floated to the top of SO for me - I had no idea you had resolved this issue so long ago, my apologies! — Adam Jenkins, Nov 06 '19 at 21:16
NP this happens when there is some activity in any question like someone else answered @Adam — Black Mamba, Nov 07 '19 at 02:20

score 0 · Answer 1 · answered May 06 '19 at 09:59

0

Was going to create a new plugin but @Anirudh helped me find this package in the comment

https://www.npmjs.com/package/mongo-escape

To use it, just before your insert query escape you object using the above package.

var escapeForMongo = require('mongo-escape').escape;
userInput = escapeForMongo({
  'foo': 'bar',
  'ba.z': {
    '$in': 'quz'
  }
})

answered May 06 '19 at 09:59

Black Mamba

13,632
6
82
105

this library replaces all the dollars($) in keys and data with escaped characters. We just need to escape keys with dollar prefix in json. – Bharat D Bhadresha Nov 04 '19 at 12:09
No idea bud but I think if that's the case then you can iterate using `Object.keys(yourObject).map(i => newObject[i.replace(/^\$/)] = yourObject[i];)` Not sure if it'll work just wrote without testing you can have something like this – Black Mamba Nov 04 '19 at 12:47
Object.keys won't work for nested keys like your example above. – Bharat D Bhadresha Nov 04 '19 at 13:24
Hmm it's gonna be something iterative. I'll see either I can find some – Black Mamba Nov 04 '19 at 15:07
https://stackoverflow.com/a/19323734/6517383 You can use this @BharatDBhadresha – Black Mamba Nov 05 '19 at 09:51

score 0 · Answer 2 · edited Nov 05 '19 at 12:47

function escapeMongo(obj) {
    return JSON.parse(
        JSON.stringify(obj),
        function(p, v) {
            if ((p[0] === '$') || (p.includes('.'))) {
                this[p.replace(/\$/g, '\uFF04').replace(/\./g, '\uFF0E')] = v;
                return;
            }
            return v;
        }
    )
}

userInput = {
    'foo': 'bar',
    'ba.z': {
        '$in': 'quz'
    }
}
console.log(userInput)
console.log(escapeMongo(userInput))

You can use JSON.parse to process a JSON string before creating object.

How to escape $ while inserting inside mongoDB

2 Answers2