How to filter out the documents based on dynamic keys before using $objectToArray?

I need to group based on dynamic keys. So I’m using $objectToArray to query on dynamic keys.

But I need to filter out the documents based on a condition so as to reduce the input going to $objectToArray. Because I have millions of documents and I just want only a subset of the object fields to be fed to the $objectToArray operator.

My aim is to get better query performance by reducing the amount of data passed to $objectToArray operator

A sample format of my MongoDB schema:

[
  {
    "fields": {
      "field_1": {              /* dynamic key */
        "name": "f1",
        "first": {
          "check": true
        }
      },
      "field_2": {             /* dynamic key */
        "name": "f2",
        "second": {
          "check": true
        }
      },
      "description": "abc",
      "summary": {
        "val": "xyz"
      }
    }
  },
  {
    "fields": {
      "field_1": {           /* dynamic key */
        "name": "f1",
        "second": {
          "check": false
        }
      },
      "field_2": null,      /* dynamic key */
      "field_3": {         /* dynamic key */
        "name": "f3",
        "second": {
          "check": true
        },
        "first": {
          "check": true
        }
      },
      "description": "lmn",
      "summary": {
        "val": "abc"
      }
    }
  }
]

There are also many fields other than dynamic fields.

I need to filter out the documents before being fed to $objectToArray operator based on the following condition:

  1. fields.<*dynamicKey*>.first exists
    OR
  2. fields.<*dynamicKey*>.second exists

Expected output:

[
  {
    "fields": {
      "field_1": {
        "name": "f1",
        "first": {
          "check": true
        }
      },
      "field_2": {
        "name": "f2",
        "second": {
          "check": true
        }
      }
    }
  },
  {
    "fields": {
      "field_1": {
        "name": "f1",
        "second": {
          "check": false
        }
      },
      "field_3": {
        "name": "f3",
        "second": {
          "check": true
        },
        "first": {
          "check": true
        }
      }
    }
  }
]

How can I achieve this use case without changing my document structure?

db.collection.aggregate([
  {
I need a stage to filter out documents here
  },
  {
    $project: {
      data: {
        $objectToArray: "$fields"
      }
    }
  }

My aggregation query to group based on dynamic keys:

Hello

I think you ask something similar to this

If the keys are uknown,the only way that i know is javascript and $function operator.

Upvote that Jira(see second link) if you want also,but i think that jira is only for get($$doc,$$key),i tihnk we need remove also at least

3 Likes

I played with this for a bit and wasn’t able to figure out a way to do the filtering before doing $objectToArray. In case it’s helpful, here is a pipeline I created that does the filtering after $objectToArray.

[{$project: {
  fields: {$objectToArray: "$fields"}
}}, {$addFields: {
  "shouldInclude": {
    $map: {
      input: "$fields",
      as: "myFields",
      in: { $or: [
        { $ifNull: ["$$myFields.v.first", false] },
        { $ifNull: ["$$myFields.v.second", false] }
        ]
      }
    }
  }
}}, {$addFields: {
  shouldInclude: {$anyElementTrue: ["$shouldInclude"]}
}}, {$match: {
  shouldInclude: true
}}]
2 Likes

Given that you say specifically you want to do that without changing the document structure I assume you know that this data modeling is probably not ideal. Changing the data modeling will likely improve performance. But without changing the data model, I think the below solution works. Note that it just checks if there is a field called second not that the field value of second.check is “true” - I think that is what you wanted but am not sure… If that was not what you wanted it can be easily changed though. :slight_smile:

Note that you need to be using MongoDB 4.4 for this to work because it uses custom aggregation expressions

db.collection.aggregate(
    [{
      $addFields: {
        has_first_or_second: {
          $function: {
            body: function(fields) {
              for (var f in fields) {
                if (Object.prototype.hasOwnProperty.call(fields[f], 'first')) {
                  return true;
                }
                if (Object.prototype.hasOwnProperty.call(fields[f], 'second')) {
                  return true;
                }
              }
              return false;
            },
            args: ["$fields"],
            lang: "js"
          }
        }
      }
    }, 
    {
      $match: {has_first_or_second: true}
    },
    {
      $project: {has_first_or_second: 0}
    }
    ])
2 Likes

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.