Recover corrupted files (MongoDB 3.0.12) .wt

Hi
we had an issue with one of our legacy MongoDB server
that was running on single instance ,
The server had unexpected shut down (power issue) and fail to start after
is there a way we can recover the data from the files , as we cannot bring back the server
we try reover etc … no success

[initandlisten] Assertion: 28595:-31803: WT_NOTFOUND: item not found
STORAGE [initandlisten] exception in initAndListen: 28595 -31803: WT_NOTFOUND: item not found, terminating

any idea / help / suggestion / support will be appreciated

Hi @Rami_Avital

You can try mongod’s --repair option. It is very likely that the files are now corrupted and your only option is to restore from a backup.

This is one of the scenarios that running a replicaset protects against as well as regular tested backups.

1 Like

we tried but no luck …
we need some expert to try restore the data from files or some other magic

Hi @Rami_Avital,

Recovering from data file corruption is unfortunately challenging and often unsuccessful.

However, I noticed you are running a very old version of MongoDB (3.0.12 was released in May, 2016).

The repair functionality for WiredTiger has been improved in more recent server versions (4.0.3+) per SERVER-19815.

This isn’t guaranteed magic, but I would:

If the repair procedure results in a usable deployment, I expect you will either have to remain on MongoDB 4.0 or mongodump and mongorestore into your 3.0 deployment.

Also to be clear: the --repair option is a last resort for data recovery and is definitely not a substitute for proper backups. The repair process will salvage data structures that can be read and skip those that cannot. This does not guarantee you can recover all of your data, and it is likely that some data will be missing depending on the nature of the data file corruption. However, if you don’t have any recent backups this has a chance of getting your deployment back online.

If you have at least one known good backup (even if not recent), I would compare data that is expected to be present in both your backup and the repaired database.

If the newer --repair process fails, this unfortunately is the stage at which you may need to accept that you have to return to your last good backup.

A final approach would be to try heroic efforts to salvage raw data. This is rarely successful and can involve a lot of deep dive effort. @alexbevi’s article on Recovering a WiredTiger collection from a corrupt MongoDB installation may be a useful guide, but note that this article predated the enhanced repair process I suggested above.

Regards,
Stennie

1 Like