I have a requirement where I am migrating data from Kafka using Kafka-Connect and dumping the data to collection A. And, I am transforming the data present in collection A using Aggregation commands such as $project, $map and $merge, final step is to merge the transformed data from collection A to collection B.
There are around 1612790 documents in collection A (18.91GB size), now when I click on “merge documents” button only some 30K documents are being pushed to collection B and later the merge aggregation command throws error: “An unknow error occured”.
Please let me know why this error is occurring ? I am not sure what is the exact issue. I suspect it could be the size of the data which it is not able to migrate in one shot.
Kindly help me out on this issue.