We are planning to upgrade our mongodb cluster and I may need some advices . Our current setup is:
3 servers on AWS
one replica set running mongodb 3.4
one replica set running mongodb 2.6
These are managed by Cloud Manager (or at least used to be managed as mongodb 2.6 is no longer supported by Cloud Manager).
What we want to do is to “merge” these two RS into a single one running mongodb 3.6. We also want to update our OS, hardware and all.
If I had only the mongodb 3.4 RS, it would be “easy”. I would just add 3 more nodes to the RS, wait for it to synchronize and remove old servers from the RS. Then just upgrade to version 3.6 using cloud manager.
However here I also want to deal with the mongodb 2.6 RS. Not sure how can I proceed without “downtime”. Any ideas ?
What we want to do is to “merge” these two RS into a single one running mongodb 3.6.
Members of a single replica set need to share a common history of changes via the oplog. For members of two distinct replica sets the only way to merge data would be dumping & restoring data from the replica set you want to retire into the replica set you want to keep.
You could backup using mongodump & mongorestore, but may need something more bespoke depending on the overlap of data between your two replica sets. For example, if both replica sets have common databases you may want to merge, rename, or ignore duplicate databases & collections. Merging data would be more straightforward if both replica sets contain different database names.
We also want to update our OS, hardware and all.
I would approach this in several stages so you don’t conflate potential issues due to hardware or O/S upgrades with your major version upgrade and replica set consolidation. Changing many variables at once may save on elapsed time, but could dramatically complicate diagnostics and troubleshooting should anything not go to plan.
For example, you could:
Provision new servers running the new O/S and latest version of MongoDB 3.4.x.
Add a 3.4 secondary on a new server to your 3.4 replica set and ensure there are no unexpected issues. You may want to prevent this secondary from becoming primary during your test period.
Add additional 3.4 secondaries on your new servers.
Promote one of the new 3.4 secondaries to primary and retire your older 3.4 secondaries.
Plan and test the replica set upgrade to MongoDB 3.6. Note: you may also need to upgrade your MongoDB drivers before upgrading your server.
Create a plan to migrate the documents from your 2.6 replica set into your 3.6 deployment.
The upgrade of your 3.4 replica set as well as the major version upgrade from 3.4 to 3.6 should have negligible impact on availability so long as you follow rolling maintenance procedures and keep a strict majority of configured voting members online during replica set upgrades.
You could perform some of these steps in a different order (such as migrating documents from 2.6 to 3.4 prior to the 3.6 upgrade) if that better suits your business requirements.
Thank you very much for your answer. That confirms what I thought, I will need to schedule some maintenance time for the apps running on the 2.6 RS. Databases between the 2 RS do not overlap so that’s already a good thing !
. Our current setup is:
!