We are experiencing major issues while inserting high volume of documents into Mongo DB. Yesterday we published 12 GB of data in one hour which is the volume we are going to receive on Production environment. So we are wondering why we are facing errors.
And also we have noticed loss of messages. 553,430 documents should be inserted but we can see only 254024 documents got inserted.
We have an M50 cluster.
Connection URL configured in our Dataflow Job (with full hostnames, not the shortened versions here):
mongodb://<user>:<password>@cluster-shard-00-00:27017,cluster-shard-00-01:27017,cluster-shard-00-02:27017/test?ssl=true&replicaSet=<clustername>&authSource=admin&retryWrites=true&w=majority
If you need any more details or about our data flow job which is inserting the data please do let us know.