Document constructed by $facet is 104857822 bytes, which exceeds the limit of 104857600 bytes

Working with Data
1

My M10 cluster is complaining about exicding a memory limit. My collection looks like this:

    { "_id" : ObjectId("6024656264c97c56e0705cd6"), "company_id" : ObjectId("602454847756575710020545"), "email" : "xxxxx@gmail.co" }
    { "_id" : ObjectId("6024656264c97c56e0705cd7"), "company_id" : ObjectId("602454847756575710020545"), "email" : "xxxxx@gmail.com" }

And I am running this aggregation:

    [
        {
            "$facet": {
                "emails": [
                    {
                        "$project": {
                            _id: false,
                            "k": "$email",
                            "v": "1"
                        }
                    }
                ]
            }
        },
        {
            "$project": {
                "emails": {
                    $arrayToObject: "$emails"
                }
            }
        }
    ]

I understand the 100M state limit, but shouldn’t allowDiskUse:true address that? I am using mongoose and am passing that parameter like so:

Collection.aggreagate([…]).allowDiskUse(true).then(result => {…

This is a similar post here that seems to suggest that the issue has to do with the 16MB doc limit. Why, then, does is mention the “the limit of 104857600 bytes” which is about 100MB? I don’t see it complaining about 16 MB anywhere.

2

Hi Zarif, welcome to the community!

In addition to the result size, there is a 100 MB limit on each stage.

Have you checked the profiler or diagnostic logs to confirm whether allowDiskUse is being correctly set by Mongoose?

3

Hey Andrew,

The reason I believe Mongoose properly sets allowDiskUse to true is because I had first attempted to write that same aggregation using a somewhat different pipeline which involved $push inside a $group. Without passing allowDiskUse:true, I got this error:

Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.

When I set the flag to true like so:

Collection.aggreagate([…]).allowDiskUse(true).then(result => {…

and rerun the aggregation, the error turns into:

$push used too much memory and cannot spill to disk. Memory limit: 104857600 bytes

So clearly Atlas sees my passing the flag, but somehow is unable to do anything with it. This makes me think there is some setting in my Atlas panel that I need to flip on.

Do you have any insight on this?

4

Checking with one of our experts on the implementation of the aggregation framework.