MongoDB.live, free & fully virtual. June 9th - 10th. Register Now

How to recover data

Hi, I have a MongoDB database made of a replica set with 3 members.
For various reasons the database got corrupted and whenever I try to start any of the members, they throw various error and crash again.

What I would like to do is to save the data.

I was thinking about moving from a replica set installation to a Standalone one to simplify things, but since I can’t connect to any of the members I can’t run the Mongo Shell and change the configuration (e.g. rs.remove(“member-host”))

I have tried to run “mongo --repair” and it seems to go through the data quite happily (WiredTiger progress WT_SESSION.verify etc.), but when it’s finished and I try to start MongoDB it still doesn’t work. It gets stuck trying to recover from the OpLog e.g

2020-04-14T14:30:36.647+0100 I COMMAND  [initandlisten] command local.oplog.rs command: getMore { getMore: 29008001961, collection: "oplog.rs", $db: "local" } originatingCommand: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1584541935, 5) } }, oplogReplay: true, $db: "local" } planSummary: COLLSCAN cursorid:29008001961 keysExamined:0 docsExamined:62847 numYields:491 nreturned:62847 reslen:16777354 locks:{ Global: { acquireCount: { r: 492 } }, Database: { acquireCount: { r: 492 } }, oplog: { acquireCount: { r: 492 } } } protocol:op_msg 114ms
  1. Is there a way to get rid of the OpLog? Perhaps by deleting some specific files in the Data folder?
  2. Is there a way to start just one member of my Replica Set in “safe mode”? i.e. without trying to be cleaver, without trying to connect to the other members, without trying to use the OpLog?
  3. Is there a way I can export the datafiles into a format which I can then import into a brand new MongoDB installation? I know how to use mongoexport and mongodump, but they both require a running instance

Many thanks!

Please share some logs, it might help finding the root cause of the problem.

Best to see the log first before taking any action.

If you want to start a node standalone. Remove the replicaSet configuration/flags.
It is usually a good idea to block external connections or bind to a different port.

Once you have found an acceptable node shut it down. You can use the data files to seed a node, no need to export/dump.

Much of this is covered extensively in the documentation:
https://docs.mongodb.com/manual/tutorial/perform-maintence-on-replica-set-members/#perform-maintenance-on-replica-set-members
https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/#sync-by-copying-data-files-from-another-member

2 Likes

Thank you Chris!
I was able to start one node standalone by removing the --replSet parameter. I can’t believe I didn’t think of that: sometimes when you look at something for too long…
I am now able to connect with MongoDump and I’m exporting the only collection I really need to save.
Thank you again!