My testing shows that zstd will give me better compression as well as faster database access times, so I’d like to upgrade to 4.2+ and migrate to the new compression algorithm.
Server is an unsharded 3 member replica set running v3.6.8 CE. Database is ~4.5TB, about 1/2 of which is indexes. Data size is ~16TB. The hardware probably doesn’t matter too much, but the primary is a 96 core 512GB memory epyc server with nvme drives on software raid.
I understand the incremental upgrade process from v3.6 to v4.2+, but what is the process to change the database compression while minimizing downtime? The database is far too large for a simple dump/restore. Replica restore from scratch is extremely slow and the oplog isn’t nearly big enough. Current oplog size of 150GB only shows ~7hrs, while a full replica restore seems like it takes days if not weeks.
I’m guessing that this is going to involve some combination of speeding up the replica sync and increasing the oplog. I would appreciate any thoughts or suggestions.