Hi Team,
We are running MongoDB replica set with 1 Primary, 1 Secondary and 1 Arbiter. Secondary instance was lagging by 1000 seconds and Primary went down due to Hardware failure.
Now, Secondary become primary but data of 1000 seconds is lost. How can I recover it?
Kindly suggest how to solve it.
Cheers,
Mukesh Kumar
Hi Mukesh, if can’t get the data from the machine that failed (or lucky enough to have had a backup run just before the crash) then there is no way to retrieve that data.
Out of curiosity, was the secondary set up to be delayed? If not you might want to check why you had ~17 minutes of latency, In most cases you shouldn’t have more than a few seconds at most.
Sorry I don’t have better news for you.
Thanks @Doug_Duncan for the update,
We somehow recover the data and downed Primary node but should I add it to the replica set. We don’t have backup and how can I restore the 1000 seconds of data loss. And newly elected primary received write operation also.
Please let me know, what to do and how?
Cheers,
Mukesh Kumar
When you add the crashed PRIMARY node back into the replica set it will join as a secondary as it doesn’t have the most recent data. It will recognize that is has data that the rest of the replica set doesn’t have an will rollback that data.
Before adding the machine back to the replica set however I would go ahead and make a copy of that data just so you have it in case something goes bad. I would also make a backup of your current PRIMARY member (you should be doing backups on regular basis anyways to make sure you can restore as recently as your SLAs require).
