We have deployed Mongo DB as a Kubernetes replica set. We noticed that one of the mongodb replica set members crashed, and from there is not able to recover. It is in a crash loop.
Mongo version: 4.2.2
Number of Replica sets 3.
Issue observed in only one of the members.
The logs show below.
Mar 8, 2021 @ 02:10:07.802 2021-03-08T10:10:07.801+0000 F - [rsBackgroundSync] Invariant failure !stableTimestamp || stableTimestamp->isNull() || appliedThrough.isNull() || *stableTimestamp == appliedThrough.getTimestamp() Stable timestamp Timestamp(1615198135, 135) does not equal appliedThrough timestamp { ts: Timestamp(1615198117, 1), t: 208 } src/mongo/db/repl/replication_recovery.cpp 352
Mar 8, 2021 @ 02:10:07.802
Mar 8, 2021 @ 02:10:07.802 ***aborting after invariant() failure
Mar 8, 2021 @ 02:10:07.802
Mar 8, 2021 @ 02:10:07.802
Mar 8, 2021 @ 02:10:07.812 2021-03-08T10:10:07.812+0000 F - [rsBackgroundSync] Got signal: 6 (Aborted).
Mar 8, 2021 @ 02:10:07.812 ----- BEGIN BACKTRACE -----
Mar 8, 2021 @ 02:10:07.812 0x55f25d8c0341 0x55f25d8bfb3e 0x55f25d8bfbd6 0x7f31f2fc4890 0x7f31f2bffe97 0x7f31f2c01801 0x55f25bd96e6f 0x55f25bb194ce 0x55f25c13d1fb 0x55f25c13fb24 0x55f25c115c35 0x55f25c116fb8 0x55f25bb26a01 0x55f25c117557 0x55f25c117698 0x55f25d9e600f 0x7f31f2fb96db 0x7
… Mongo DB Version Information…
Mar 8, 2021 @ 02:10:16.735 2021-03-08T10:10:16.734+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
Mar 8, 2021 @ 02:10:16.735 2021-03-08T10:10:16.734+0000 I CONTROL [initandlisten] allocator: tcmalloc
Mar 8, 2021 @ 02:10:16.735 2021-03-08T10:10:16.734+0000 I CONTROL [initandlisten] modules: none
Mar 8, 2021 @ 02:10:16.735 2021-03-08T10:10:16.734+0000 I CONTROL [initandlisten] build environment:
Mar 8, 2021 @ 02:10:16.735 2021-03-08T10:10:16.734+0000 I CONTROL [initandlisten] db version v4.2.2
Mar 8, 2021 @ 02:10:16.735 2021-03-08T10:10:16.734+0000 I CONTROL [initandlisten] git version: a0bbbff6ada159e19298d37946ac8dc4b497eadf
Any idea of why it happened?
Thanks,
Parvathi.