Hi All,
I am getting memory size limit error while running multiple sub-pipelines in $facet. Can someone help me on this issue.
Scenario: I have a crown job which runs once a day. I want to execute a pipeline in it against a collection with millions of documents.
[
{
$facet: {
query1: [pipeline_1],
query2: [pipeline_2],
query3: [pipeline_3]
...
query_n: [pipeline_n]
},
},
{
$merge:{ into: some_collection}
}
]
I tried allowDiskUse=true, But still giving same error.
What can be the work around on this. Please help.
Hi @Pradip_Kumar,
How did you specified the allowDiskUse: true
?
AFAIK, this stage should allow it to be used.
Thanks
Pavel
I used below syntax from CLI:
db.collection_name.aggregate( [pipeline], {allowDiskUse: true}}
and from pymango i am using below syntax :
db_client.aggregate(
db_name, collection_name, pipeline, allow_disk_use=True)
It is not working in both cases.
@Pradip_Kumar,
According to PyMongo the parameter name is still called allowDiskUse
:
https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.aggregate
What I suspect is that the error is not rrelated to the 100MB per stage limit, but maybe to the output of $facet object that we get from your agg…
This object is not suppose to cross 16MB actually.
1 Like
I am running into the this same exact issue. My error is:
MongoError: document constructed by $facet is 104857822 bytes, which exceeds the limit of 104857600 bytes
I understand the 100M state limit, but shouldn’t allowDiskUse:true address that? I am using mongoose and am passing that parameter like so:
Collection.aggreagate([…]).allowDiskUse(true).then(result => {…
You seem to suggest that the issue has to do with the 16MB doc limit. Why, then, does is mention the “the limit of 104857600 bytes” which is about 100MB? I don’t see it complaining about 16 MB anywhere.
My aggregation looks like this:
[
{
"$facet": {
"emails": [
{
"$project": {
_id: false,
"k": "$email",
"v": "1"
}
}
]
}
},
{
"$project": {
"emails": {
$arrayToObject: "$emails"
}
}
}
]