Watch keynotes and sessions from MongoDB.live, our virtual developer conference.

WT_PANIC after restore of the crashed backend storage

Our hardware storage crashed last Friday and after the hardware replace the mongo-db service is not starting anymore.
We have roled back the system of the last known stand.

On a seperate VM we have attached the corrupt MongoDB to find a way to solve the corropution.
Did someone have any idea what I can do to fix this issue?

replication set was not enabled… So I can not switch.

  1. MongoDB version: 3.4.7
  2. OS version: Windows Server 2016
  3. Damaged file: WT_PANIC (file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value)
2020-06-19T11:13:47.647+0200 I CONTROL  [main] Trying to start Windows service 'MongoDB'
2020-06-19T11:13:47.648+0200 I CONTROL  [initandlisten] MongoDB starting : pid=976 port=27017 dbpath=D:/corpuls.data/databases/mongodb/db 64-bit host=VS39362EH
2020-06-19T11:13:47.648+0200 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2020-06-19T11:13:47.648+0200 I CONTROL  [initandlisten] db version v3.4.7
2020-06-19T11:13:47.648+0200 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2020-06-19T11:13:47.648+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1u-fips  22 Sep 2016
2020-06-19T11:13:47.649+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2020-06-19T11:13:47.649+0200 I CONTROL  [initandlisten] modules: none
2020-06-19T11:13:47.649+0200 I CONTROL  [initandlisten] build environment:
2020-06-19T11:13:47.649+0200 I CONTROL  [initandlisten]     distmod: 2008plus-ssl
2020-06-19T11:13:47.649+0200 I CONTROL  [initandlisten]     distarch: x86_64
2020-06-19T11:13:47.649+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2020-06-19T11:13:47.649+0200 I CONTROL  [initandlisten] options: { config: "C:\Program Files\MongoDB\3.4.7\mongod.conf", service: true, storage: { dbPath: "D:/corpuls.data/databases/mongodb/db", engine: "wiredTiger" }, systemLog: { destination: "file", logAppend: false, logRotate: "rename", path: "D:/corpuls.data/databases/mongodb/log/mongod.log", quiet: true, timeStampFormat: "iso8601-local", traceAllExceptions: false, verbosity: 0 } }
2020-06-19T11:13:47.650+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7679M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2020-06-19T11:13:47.659+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1592558027:659502][976:140705624250832], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2020-06-19T11:13:47.659+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1592558027:659502][976:140705624250832], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-06-19T11:13:47.659+0200 I -        [initandlisten] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2020-06-19T11:13:47.659+0200 I -        [initandlisten] 

***aborting after fassert() failure


2020-06-19T11:13:47.813+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)                     mongo::printStackTrace+0x43
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)            mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)            mongo::`anonymous namespace'::abruptQuit+0x85
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] ucrtbase.dll                                                                     raise+0x1e7
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] ucrtbase.dll                                                                     abort+0x31
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\assert_util.cpp(172)                            mongo::fassertFailedWithLocation+0x181
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(361)       mongo::`anonymous namespace'::mdb_handle_error+0x205
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(275)              __wt_eventv+0x376
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(317)              __wt_err+0x32
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(530)              __wt_illegal_value+0x5e
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\meta\meta_turtle.c(288)         __wt_turtle_read+0x2ab
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\meta\meta_table.c(269)          __wt_metadata_search+0x28e
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(269)        __conn_btree_config_set+0x22
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(337)        __wt_conn_btree_open+0x5c
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_dhandle.c(542)  __wt_session_get_btree+0x113
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_dhandle.c(534)  __wt_session_get_btree+0x1d5
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_dhandle.c(347)  __wt_session_get_btree_ckpt+0xc4
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\cursor\cur_file.c(567)          __wt_curfile_open+0x1dd
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_api.c(388)      __session_open_cursor_int+0x2f7
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_api.c(443)      __wt_open_cursor+0x1b
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\meta\meta_table.c(91)           __wt_metadata_cursor+0x99
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\conn\conn_api.c(2454)           wiredtiger_open+0xb09
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(265)  mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x932
2020-06-19T11:13:47.814+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(91)        mongo::`anonymous namespace'::WiredTigerFactory::create+0x12f
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\service_context_d.cpp(202)                        mongo::ServiceContextMongoD::initializeGlobalStorageEngine+0x59c
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(599)                                       mongo::`anonymous namespace'::_initAndListen+0x77b
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(841)                                       mongo::`anonymous namespace'::initAndListen+0x27
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\ntservice.cpp(560)                              mongo::ntservice::initService+0x53
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] sechost.dll                                                                      LsaFreeMemory+0x512
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] KERNEL32.DLL                                                                     BaseThreadInitThunk+0x14
2020-06-19T11:13:47.815+0200 F -        [initandlisten] Got signal: 22 (SIGABRT).
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] *** unhandled exception 0x0000000E at 0x00007FF894014C48, terminating
2020-06-19T11:13:47.815+0200 I CONTROL  [initandlisten] *** stack trace for unhandled exception:
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] KERNELBASE.dll                                                                     RaiseException+0x68
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)            mongo::`anonymous namespace'::abruptQuit+0x9d
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] ucrtbase.dll                                                                       raise+0x1e7
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] ucrtbase.dll                                                                       abort+0x31
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\assert_util.cpp(172)                            mongo::fassertFailedWithLocation+0x181
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(361)       mongo::`anonymous namespace'::mdb_handle_error+0x205
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\support\err.c(275)              __wt_eventv+0x376
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\support\err.c(317)              __wt_err+0x32
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\support\err.c(530)              __wt_illegal_value+0x5e
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\meta\meta_turtle.c(288)         __wt_turtle_read+0x2ab
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\meta\meta_table.c(269)          __wt_metadata_search+0x28e
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(269)        __conn_btree_config_set+0x22
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(337)        __wt_conn_btree_open+0x5c
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_dhandle.c(542)  __wt_session_get_btree+0x113
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_dhandle.c(534)  __wt_session_get_btree+0x1d5
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_dhandle.c(347)  __wt_session_get_btree_ckpt+0xc4
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\cursor\cur_file.c(567)          __wt_curfile_open+0x1dd
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_api.c(388)      __session_open_cursor_int+0x2f7
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_api.c(443)      __wt_open_cursor+0x1b
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\meta\meta_table.c(91)           __wt_metadata_cursor+0x99
2020-06-19T11:13:47.870+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\conn\conn_api.c(2454)           wiredtiger_open+0xb09
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(265)  mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x932
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(91)        mongo::`anonymous namespace'::WiredTigerFactory::create+0x12f
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\service_context_d.cpp(202)                        mongo::ServiceContextMongoD::initializeGlobalStorageEngine+0x59c
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(599)                                       mongo::`anonymous namespace'::_initAndListen+0x77b
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(841)                                       mongo::`anonymous namespace'::initAndListen+0x27
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\ntservice.cpp(560)                              mongo::ntservice::initService+0x53
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] sechost.dll                                                                        LsaFreeMemory+0x512
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] KERNEL32.DLL                                                                       BaseThreadInitThunk+0x14
2020-06-19T11:13:47.871+0200 I -        [initandlisten] 
2020-06-19T11:13:47.871+0200 I CONTROL  [initandlisten] writing minidump diagnostic file C:\Program Files\MongoDB\3.2020-06-19T09-13-47.mdmp
2020-06-19T11:13:47.921+0200 I CONTROL  [initandlisten] *** immediate exit due to unhandled exception

Welcome @Jerome_Bose

It looks like your failure resulted in the corruption of the data files. Unfortunately you are very unlikely to recover from this scenario without using a backup.

You can attempt a repair. But do not be surprised it this does not work, or recover all data.

Hi Chris,

Ok I can do this. Should I upgrade the mongoDB first to 4.x? Or should try with the current installed 3.4 based installation.

Use your existing installation.