Merge Aggregation command in Mongo Shell throws error

Working with Data
1

Hi,

I have a requirement where I am migrating data from Kafka using Kafka-Connect and dumping the data to collection A. And, I am transforming the data present in collection A using Aggregation commands such as $project, $map and $merge, final step is to merge the transformed data from collection A to collection B.

There are around 1612790 documents in collection A (18.91GB size), now when I click on “merge documents” button only some 30K documents are being pushed to collection B and later the merge aggregation command throws error: “An unknow error occured”.

Please let me know why this error is occurring ? I am not sure what is the exact issue. I suspect it could be the size of the data which it is not able to migrate in one shot.

Kindly help me out on this issue.

Thanks,
Vinay

2

Hi @Vinay_Gangaraj,

When you say pressing a button named “merge documents” are you referring to running it via compass or atlas data explorer?

If yes I assume that there are 2 limitations in place:

  1. Aggregation builder samples data to show the pipeline and therefore it might error out on a very large amount of documents like you said.
  2. There is possible a driver timeout for this operation and since it is long it might error out. I recommend looking into chrome developer console for better errors.

What I would recommend is to take the pipeline and use it in a mongo shell or any language script like python or js.

Thanks
Pavel