More info on MongoDB initial sync architecture

Hi,

I’m looking for a bit more information on the initial sync architecture with respect to the Oplog and data changes during the data copy phase.

I’m referring to this page; https://docs.mongodb.com/manual/core/replica-set-sync/

If I understand this correctly, while the data copy is happening from the source to the target, any data changes picked up on the source are written to the local database on the target. Once the data copy is complete, the changes in the target local are then executed to ensure the source and target are in sync? This is the behaviour from 3.4 onwards.

The Oplog is 150GB. My thinking is that if we generated 300GB of data changes, they would be synced to the local on target and then replayed once the data copy stage has completed?

I was always under the impression that the data copy phase had to complete before the Oplog records were overwritten. This may have been the behaviour in 3.2 and earlier. Or it could be that I never properly understood this area!

The reason for asking is that I am syncing a very large database. The replication oplog window for this replicaset can vary between 1 and 4 hours - it is a very volatile database in terms of collection’s being created and dropped very frequently.

Thanks,
Clive

That is what I remember too. 3.2 docs for reference:
https://docs.mongodb.com/v3.2/core/replica-set-sync/

Make the oplog on the new replica 450GB if you have the space. No worries!

Thanks Chris.

An increase in the Oplog size is something being considered while as a short term solution until we can look to add sharding into the mix - an 8.5TB shard isn’t particularly fun.

Thanks for the link - I didn’t think to read up on 3.2 and confirm what I thought may be the case.

Clive

Hi @clivestrong,

I’d like to directly answer some of your questions:

If I understand this correctly, while the data copy is happening from the source to the target, any data changes picked up on the source are written to the local database on the target. Once the data copy is complete, the changes in the target local are then executed to ensure the source and target are in sync? This is the behaviour from 3.4 onwards.

Correct.

The Oplog is 150GB. My thinking is that if we generated 300GB of data changes, they would be synced to the local on target and then replayed once the data copy stage has completed?

Correct.

Previously in MongoDB pre-3.4, initial sync happens in two distinct phases: data copy -> oplog pull & apply. The primary would need to have enough oplog to cover the whole initial sync time. In some cases, this could be hard to approximate.

Starting from 3.4, the oplog was tailed during the data copy phase, eliminating the need for the primary to have a large oplog to cover the data copy phase. However, the primary would still need to have oplog coverage during the oplog apply phase. This is typically a much smaller window than the data copy phase, though.

Further, in the latest MongoDB 4.4, there are further improvements in replication (see the 4.4 release notes); namely, initial sync would automatically resume in the face of transient network error, streaming replication, and the ability to set the minimum oplog retention period in hours.

Best regards,
Kevin

2 Likes

Hi Kevin,

Thanks for this - I’d pretty much confirmed my thoughts whilst running through the logs / monitoring the process but appreciate your confirming this.

I’ve read up on the 4.4 features and these are of value to us. We have some software on 4.2 and other software on 3.6 but will be looking at 4.4 at some point next year!

Clive

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.