Slow Replica Set

Ops and Admin Replication
,
1

Hi all,
I would like to share an issue that I have with my MongoDB replica set (MongoDB Community edition, 4.2.3)

I have a cluster with 3 nodes, with this hardware (two of them are virtual machines running on Vmware):

  1. First node has 4vcpu, 8 GB of RAM and 2 TB of disk
  2. Second node has 6vcpu, 8 GB of RAM and 2 TB of disk
  3. The third node is a dedicated server with an Intel Xeon E2274G (4C/8T) , 32 GB of RAM and 10 TB of disk

I have a GridFS collection with around 1,8 TB of documents

To reclaim space I removed my secondary node, I have recreated it and I have added it again to the replica set. Now MongoDB is copying documents from master to the replica.

The problem is that is very slow. After 12 hours it has copied only 36% of my collection. I have tested this operation three times and usually the entire copy ended in 6/7 hours.

In log files I see this warnings, but I don’t know what they mean

2020-03-08T11:23:29.306+0100 I CONTROL [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: config.system.sessions does not exist

Anyone have ideas?

Thanks in advance,
Jack

2

Hi @jack, welcome!

You can try creating a new member using seed data from another member. Restart the machine with a copy of a recent data directory from another member in the replica set. This procedure can replace the data more quickly but requires more manual steps. See also Replica Set Resync by Copying.

When syncing a member, choose a time when the system has the bandwidth to move a large amount of data. Schedule the synchronization during a time of low usage or during a maintenance window.

Alternatively, if you have MongoDB backup files see also Restore a Replica Set from MongoDB Backups.

Regards,
Wan.