DummyD3 is node3 of 3 node replication cluster. DummyD2 is primary and DummyD1 is Secondary (which is in sync with primary).
I tried deleting the data under data path and did rsync from other secondary node back to back twice to copy the updates while 1st was running in 2nd rsycn and started the deamon.
rsync -asvh Dummy1:/mongodb/data/db7975/* /mongodb/data/db7975
rsync -asvh Dummy1:/mongodb/data/db7975/* /mongodb/data/db7975
rm /mongodb/data/db7975/mongod.lock
rm /mongodb/data/db7975/mongod.pid
ulimit -n 65000
su - mongo7975 -c "/usr/bin/mongod --config /etc/mongo/db7975/mongodb.conf &"
echo "Rebuild is done"
Ran above command in screen from problematic node.
~# mongod --version
db version v4.0.4
I’m here looking for solution that if some has solution to help me fixing this issue.
root@DummyD3:~# tail -600f /mongodb/files/errorlogs/db7975/mongodb.log 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] MongoDB starting : pid=9285 port=7975 dbpath=/mongodb/data/db7975 64-bit host=DummyD3 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] db version v4.0.4 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] git version: f288a3bdf201007f3693c58e140056adf8b04839 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] allocator: tcmalloc 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] modules: none 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] build environment: 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] distmod: ubuntu1604 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] distarch: x86_64 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] target_arch: x86_64 2021-03-08T08:25:36.412+0000 I CONTROL [initandlisten] options: { config: "/etc/mongo/db7975/mongodb.conf", net: { bindIp: "127.0.0.1,DummyD3.domain.company.com", port: 7975, ssl: { CAFile: "/mongodb/data/keys/server/ca_chain.crt", PEMKeyFile: "/mongodb/data/keys/server/server.pem", allowConnectionsWithoutCertificates: true, mode: "requireSSL" } }, operationProfiling: { mode: "slowOp", slowOpThresholdMs: 1000 }, processManagement: { fork: true, pidFilePath: "/mongodb/data/db7975/mongo7975.pid" }, replication: { oplogSizeMB: 2048, replSet: "mongo7975" }, security: { keyFile: "/mongodb/data/keyfile/mongo_keyfile_db7975" }, storage: { dbPath: "/mongodb/data/db7975", directoryPerDB: true, mmapv1: { smallFiles: true }, wiredTiger: { engineConfig: { cacheSizeGB: 1.0 } } }, systemLog: { destination: "file", logAppend: true, path: "/mongodb/files/errorlogs/db7975/mongodb.log", quiet: true } } 2021-03-08T08:25:36.414+0000 I STORAGE [initandlisten] Detected data files in /mongodb/data/db7975 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2021-03-08T08:25:36.414+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1024M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2021-03-08T08:25:36.954+0000 E STORAGE [initandlisten] WiredTiger error (-31803) [1615191936:954630][9285:0x7fa296bfaa40], txn-recover:
__wt_txn_recover, 740: Recovery failed: WT_NOTFOUND: item not found Raw: [1615191936:954630][9285:0x7fa296bfaa40], txn-recover:
__wt_txn_recover, 740: Recovery failed: WT_NOTFOUND: item not found 2021-03-08T08:25:36.954+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191936:954737][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 384: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1615191936:954737][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 384: cache server: exiting with 1 pages in memory and 0 pages evicted 2021-03-08T08:25:36.954+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191936:954749][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 389: cache server: exiting with 278 image bytes in memory Raw: [1615191936:954749][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 389: cache server: exiting with 278 image bytes in memory 2021-03-08T08:25:36.954+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191936:954755][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 393: cache server: exiting with 886 bytes in memory Raw: [1615191936:954755][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 393: cache server: exiting with 886 bytes in memory 2021-03-08T08:25:36.991+0000 E STORAGE [initandlisten] WiredTiger error (-31803) [1615191936:991847][9285:0x7fa296bfaa40], txn-recover:
__wt_txn_recover, 740: Recovery failed: WT_NOTFOUND: item not found Raw: [1615191936:991847][9285:0x7fa296bfaa40], txn-recover:
__wt_txn_recover, 740: Recovery failed: WT_NOTFOUND: item not found 2021-03-08T08:25:36.991+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191936:991945][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 384: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1615191936:991945][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 384: cache server: exiting with 1 pages in memory and 0 pages evicted 2021-03-08T08:25:36.991+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191936:991964][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 389: cache server: exiting with 278 image bytes in memory Raw: [1615191936:991964][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 389: cache server: exiting with 278 image bytes in memory 2021-03-08T08:25:36.991+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191936:991970][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 393: cache server: exiting with 886 bytes in memory Raw: [1615191936:991970][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 393: cache server: exiting with 886 bytes in memory 2021-03-08T08:25:37.012+0000 E STORAGE [initandlisten] WiredTiger error (-31803) [1615191937:12370][9285:0x7fa296bfaa40], txn-recover:
__wt_txn_recover, 740: Recovery failed: WT_NOTFOUND: item not found Raw: [1615191937:12370][9285:0x7fa296bfaa40], txn-recover:
__wt_txn_recover, 740: Recovery failed: WT_NOTFOUND: item not found 2021-03-08T08:25:37.012+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191937:12486][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 384: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1615191937:12486][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 384: cache server: exiting with 1 pages in memory and 0 pages evicted 2021-03-08T08:25:37.012+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191937:12503][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 389: cache server: exiting with 278 image bytes in memory Raw: [1615191937:12503][9285:0x7fa296bfaa40], connection:
__wt_cache_destroy, 389: cache server: exiting with 278 image bytes in memory 2021-03-08T08:25:37.012+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615191937:12510][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 393: cache server: exiting with 886 bytes in memory Raw: [1615191937:12510][9285:0x7fa296bfaa40], connection: __wt_cache_destroy, 393: cache server: exiting with 886 bytes in memory 2021-03-08T08:25:37.013+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2021-03-08T08:25:37.013+0000 F - [initandlisten] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 65 2021-03-08T08:25:37.013+0000 F - [initandlisten]
***aborting after fassert() failure
2021-03-08T08:25:37.023+0000 F - [initandlisten] Got signal: 6 (Aborted). 0x55c2e748caa1 0x55c2e748bcb9 0x55c2e748c19d 0x7fa295382390 0x7fa294fdc428 0x7fa294fde02a 0x55c2e5ac1ea7 0x55c2e5bb8eb3 0x55c2e5b90161 0x55c2e5b9611f 0x55c2e5b78149 0x55c2e62613a2 0x55c2e5a45251 0x55c2e5b2d94b 0x55c2e5ac3a39 0x7fa294fc7830 0x55c2e5b2bab9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55C2E50B2000","o":"23DAAA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55C2E50B2000","o":"23D9CB9"},{"b":"55C2E50B2000","o":"23DA19D"},{"b":"7FA295371000","o":"11390"},{"b":"7FA294FA7000","o":"35428","s":"gsignal"},{"b":"7FA294FA7000","o":"3702A","s":"abort"},{"b":"55C2E50B2000","o":"A0FEA7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55C2E50B2000","o":"B06EB3","s":"_ZN5mongo17wtRCToStatus_slowEiPKc"},{"b":"55C2E50B2000","o":"ADE161","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_"},{"b":"55C2E50B2000","o":"AE411F","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55C2E50B2000","o":"AC6149"},{"b":"55C2E50B2000","o":"11AF3A2","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},{"b":"55C2E50B2000","o":"993251"},{"b":"55C2E50B2000","o":"A7B94B","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55C2E50B2000","o":"A11A39","s":"main"},{"b":"7FA294FA7000","o":"20830","s":"__libc_start_main"},{"b":"55C2E50B2000","o":"A79AB9","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.0.4", "gitVersion" : "f288a3bdf201007f3693c58e140056adf8b04839", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-54-generic", "version" : "#58~16.04.1-Ubuntu SMP Mon Jun 24 13:21:41 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55C2E50B2000", "elfType" : 3, "buildId" : "955496FB361B86CAB5A87FD0A0E11190E01CE1BF" }, { "b" : "7FFD916D3000", "elfType" : 3, "buildId" : "0D1E778D0972C90019EB42B2CDA255878DD87639" }, { "b" : "7FA296782000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "5C1A06A89F89E1ADAAA507BC5580C0A7931B0AB2" }, { "b" : "7FA296567000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, { "b" : "7FA296122000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, { "b" : "7FA295EB9000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, { "b" : "7FA295CB5000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7FA295AAD000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7FA2957A4000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7FA29558E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FA295371000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7FA294FA7000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7FA2969F1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" }, { "b" : "7FA294D74000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, { "b" : "7FA294B58000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7FA29490E000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "41971A4A3CCDC54A447F41DF4BD96C948C546E0E" }, { "b" : "7FA2946FF000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "8E613D0B8D8E3537785637424782BE8502ABABD2" }, { "b" : "7FA2944AE000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3890D33727391E4A85DC0F819AB0AA29BB5DFC86" }, { "b" : "7FA294294000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7FA293F64000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "17285B5F2BCC671E0A7FA3E29CCD143509B648CD" }, { "b" : "7FA293D31000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7FA293AFB000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7FA29387B000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7FA2935A9000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "0EEF7058B0737B68BDF89E5DC604D0AC389C8BB1" }, { "b" : "7FA29337A000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "FFBA483A43D9EF73925AC116811890C037523DA1" }, { "b" : "7FA293176000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7FA292F6B000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "B789D8D4B4FC333405AB34387D9237F954060EA4" }, { "b" : "7FA292D50000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "87783DF8A1058CD150F8886CB36340384093C18F" }, { "b" : "7FA292B0F000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7FA2928AB000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7FA292698000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, { "b" : "7FA292494000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7FA29228B000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7FA292001000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7FA291D5F000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7FA291B2C000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7FA291916000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7FA29170E000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7FA2914E5000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7FA2912D6000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7FA29108B000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7FA290DB6000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "3B0454E57467057071F7AD49651E0FA7B01CF5C7" }, { "b" : "7FA290B7E000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "FD61CA7A6D603E94E5EFD5C88D8810AE104FCF40" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55c2e748caa1]
mongod(+0x23D9CB9) [0x55c2e748bcb9]
mongod(+0x23DA19D) [0x55c2e748c19d]
libpthread.so.0(+0x11390) [0x7fa295382390]
libc.so.6(gsignal+0x38) [0x7fa294fdc428]
libc.so.6(abort+0x16A) [0x7fa294fde02a]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c2e5ac1ea7]
mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x413) [0x55c2e5bb8eb3]
mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0x2D1) [0x55c2e5b90161]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x63F) [0x55c2e5b9611f]
mongod(+0xAC6149) [0x55c2e5b78149]
mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x552) [0x55c2e62613a2]
mongod(+0x993251) [0x55c2e5a45251]
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x10DB) [0x55c2e5b2d94b]
mongod(main+0x9) [0x55c2e5ac3a39]