Election fails although 3 nodes are available!

So totally I have 7 replica members. 4 in SERVER1 and 3 in SERVER2. Due to some issue, SERVER1 has shutdown. Although 3 members still remain in SERVER2 (2S-1A) election is not taking place and I am not sure why.

This is the rs.status() in the secondary in SERVER2

{
        "set" : "REPLICASETNAME",
        "date" : ISODate("2021-01-18T06:32:58.873Z"),
        "myState" : 2,
        "term" : NumberLong(282),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "majorityVoteCount" : 4,
        "writeMajorityCount" : 4,
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "lastCommittedWallTime" : ISODate("2021-01-18T03:37:14.660Z"),
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "readConcernMajorityWallTime" : ISODate("2021-01-18T03:37:14.660Z"),
                "appliedOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "lastAppliedWallTime" : ISODate("2021-01-18T03:37:14.660Z"),
                "lastDurableWallTime" : ISODate("2021-01-18T03:37:14.660Z")
        },
        "lastStableRecoveryTimestamp" : Timestamp(1610941034, 1),
        "lastStableCheckpointTimestamp" : Timestamp(1610941034, 1),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "SERVER1:27017",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.494Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 1,
                        "name" : "SERVER1:27018",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.493Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 2,
                        "name" : "SERVER1:27019",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.494Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 3,
                        "name" : "SERVER1:27020",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.493Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 4,
                        "name" : "SERVER2:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 563,
                        "optime" : {
                                "ts" : Timestamp(1610941034, 1),
                                "t" : NumberLong(282)
                        },
                        "optimeDate" : ISODate("2021-01-18T03:37:14Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "could not find member to sync from",
                        "configVersion" : 80072,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 5,
                        "name" : "server2:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 441,
                        "optime" : {
                                "ts" : Timestamp(1610941034, 1),
                                "t" : NumberLong(282)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1610941034, 1),
                                "t" : NumberLong(282)
                        },
                        "optimeDate" : ISODate("2021-01-18T03:37:14Z"),
                        "optimeDurableDate" : ISODate("2021-01-18T03:37:14Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:58.568Z"),
                        "lastHeartbeatRecv" : ISODate("2021-01-18T06:32:58.653Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 80072
                },
                {
                        "_id" : 6,
                        "name" : "SERVER2:27020",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 441,
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:58.546Z"),
                        "lastHeartbeatRecv" : ISODate("2021-01-18T06:32:58.087Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 80072
                }
        ],
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1610941034, 1),
                "signature" : {
                        "hash" : BinData(0,"JnS/QEMfZgZaDZezG44AVJ5yod4="),
                        "keyId" : NumberLong("6859706153517973505")
                }
        },
        "operationTime" : Timestamp(1610941034, 1)
}

This is the rs.conf()

{
        "_id" : "REPLICASETNAME",
        "version" : 68,
        "protocolVersion" : NumberLong(1),
        "writeConcernMajorityJournalDefault" : true,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "SERVER1:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 10,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "SERVER1:27018",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 5,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "SERVER1:27019",
                        "arbiterOnly" : true,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 0,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 3,
                        "host" : "SERVER1:27020",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 5,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 4,
                        "host" : "SERVER2:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 5,
                        "host" : "SERVER2:27018",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 6,
                        "host" : "SERVER2:27020",
                        "arbiterOnly" : true,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 0,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5e450995dc745aa0d45e8d74")
        }
}

The logs of both SERVER2 secondary 1 and 2 looks like this

2021-01-18T06:36:26.998+0000 I  ELECTION [replexec-7] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:36:36.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27020
2021-01-18T06:36:36.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27018
2021-01-18T06:36:36.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27017
2021-01-18T06:36:36.994+0000 I  CONNPOOL [Replication] Connecting to SERVER1:27019
2021-01-18T06:36:37.192+0000 I  ELECTION [replexec-7] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:36:48.172+0000 I  ELECTION [replexec-7] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:36:56.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27018
2021-01-18T06:36:56.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27017
2021-01-18T06:36:56.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27020
2021-01-18T06:36:56.994+0000 I  CONNPOOL [Replication] Connecting to SERVER1:27019
2021-01-18T06:36:59.118+0000 I  ELECTION [replexec-6] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:37:06.544+0000 I  NETWORK  [conn173] end connection 127.0.0.1:34860 (156 connections now open)
2021-01-18T06:37:09.613+0000 I  ELECTION [replexec-6] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:37:16.159+0000 I  NETWORK  [listener] connection accepted from 172.28.0.1:47278 #174 (157 connections now open)
2021-01-18T06:37:16.160+0000 I  NETWORK  [conn174] received client metadata from 172.28.0.1:47278 conn174: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.4.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "18.04" } }
2021-01-18T06:37:16.190+0000 I  ACCESS   [conn174] Successfully authenticated as principal vvgcaameig on admin from client 172.28.0.1:47278
2021-01-18T06:37:16.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27017
2021-01-18T06:37:16.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27020
2021-01-18T06:37:16.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27018
2021-01-18T06:37:16.994+0000 I  CONNPOOL [Replication] Connecting to SERVER1:27019
2021-01-18T06:37:19.924+0000 I  ELECTION [replexec-6] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:37:24.291+0000 I  NETWORK  [conn174] end connection 172.28.0.1:47278 (156 connections now open)
2021-01-18T06:37:30.959+0000 I  ELECTION [replexec-7] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:37:36.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27020
2021-01-18T06:37:36.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27018
2021-01-18T06:37:36.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27017
2021-01-18T06:37:36.994+0000 I  CONNPOOL [Replication] Connecting to SERVER1:27019
2021-01-18T06:37:38.720+0000 I  NETWORK  [listener] connection accepted from SERVER2IP:31560 #175 (157 connections now open)
2021-01-18T06:37:38.720+0000 I  NETWORK  [conn175] received client metadata from SERVER2IP:31560 conn175: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.4.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "18.04" } }
2021-01-18T06:37:38.757+0000 I  ACCESS   [conn175] Successfully authenticated as principal vvgcaameig on admin from client SERVER2IP:31560
2021-01-18T06:37:41.175+0000 I  ELECTION [replexec-6] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:37:51.868+0000 I  ELECTION [replexec-7] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2021-01-18T06:37:56.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27018
2021-01-18T06:37:56.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27017
2021-01-18T06:37:56.648+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to SERVER1:27020
2021-01-18T06:37:56.994+0000 I  CONNPOOL [Replication] Connecting to SERVER1:27019












{
        "set" : "REPLICASETNAME",
        "date" : ISODate("2021-01-18T06:32:58.873Z"),
        "myState" : 2,
        "term" : NumberLong(282),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "majorityVoteCount" : 4,
        "writeMajorityCount" : 4,
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "lastCommittedWallTime" : ISODate("2021-01-18T03:37:14.660Z"),
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "readConcernMajorityWallTime" : ISODate("2021-01-18T03:37:14.660Z"),
                "appliedOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1610941034, 1),
                        "t" : NumberLong(282)
                },
                "lastAppliedWallTime" : ISODate("2021-01-18T03:37:14.660Z"),
                "lastDurableWallTime" : ISODate("2021-01-18T03:37:14.660Z")
        },
        "lastStableRecoveryTimestamp" : Timestamp(1610941034, 1),
        "lastStableCheckpointTimestamp" : Timestamp(1610941034, 1),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "SERVER1:27017",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.494Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 1,
                        "name" : "SERVER1:27018",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.493Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 2,
                        "name" : "SERVER1:27019",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.494Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 3,
                        "name" : "SERVER1:27020",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:57.493Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 4,
                        "name" : "SERVER2:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 563,
                        "optime" : {
                                "ts" : Timestamp(1610941034, 1),
                                "t" : NumberLong(282)
                        },
                        "optimeDate" : ISODate("2021-01-18T03:37:14Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "could not find member to sync from",
                        "configVersion" : 80072,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 5,
                        "name" : "dev.instasafe.io:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 441,
                        "optime" : {
                                "ts" : Timestamp(1610941034, 1),
                                "t" : NumberLong(282)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1610941034, 1),
                                "t" : NumberLong(282)
                        },
                        "optimeDate" : ISODate("2021-01-18T03:37:14Z"),
                        "optimeDurableDate" : ISODate("2021-01-18T03:37:14Z"),
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:58.568Z"),
                        "lastHeartbeatRecv" : ISODate("2021-01-18T06:32:58.653Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 80072
                },
                {
                        "_id" : 6,
                        "name" : "dev.instasafe.io:27020",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 441,
                        "lastHeartbeat" : ISODate("2021-01-18T06:32:58.546Z"),
                        "lastHeartbeatRecv" : ISODate("2021-01-18T06:32:58.087Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 80072
                }
        ],
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1610941034, 1),
                "signature" : {
                        "hash" : BinData(0,"JnS/QEMfZgZaDZezG44AVJ5yod4="),
                        "keyId" : NumberLong("6859706153517973505")
                }
        },
        "operationTime" : Timestamp(1610941034, 1)
}

As per the logs it keeps trying to connect to SERVER1 although it is down. Why not try SERVER2 members itself? There is 2 Secondaries and 1 Arbiter and it can easily do an election

Hello @Shrinidhi_Rao,

In a replic-set, in case a primary node goes down, one of the secondary nodes take its place, as primary; this process is called as failover. The failover mechanism also means that a majority of the nodes must be available for a primary to be elected. The available nodes decide which node becomes a primary, thru the process of election.

In a seven member replica-set (with all members eligible to vote) the majority is 4. There must be majority voting members for an election to take place and elect a new primary.

In your case, without the 4 voting members (of SERVER1) the election is not possible with just the remaining 3 members to elect a new primary.

2 Likes

What @Prasad_Saya said is correct: you need to be able to reach a majority of the voting members in a Replica Set to be able to trigger an election.

Extreme example: you have 50 nodes in your RS. Only 7 voting members in it (maximum). If you are unlucky and 4 of these voting members go down, you will be stuck with 46 nodes that can’t elect a Primary unless you reconfigure the Replica Set.

You can read more about elections and failover operation in MongoDB’s documentation.

Also, I wanted to ask you why you had 4 nodes on a single node and 3 on another node. That’s not a production ready setup because it’s breaks the first principale of the Replica Sets which is High Availability. This is the #1 reason for RS to exist. That’s why we usually recommend for production clusters to put the different node in 3 different data centers which are completely independent from one to another (or at least different regions if you are in a cloud provider like eu-west-1, eu-west-2 and eu-west-3).

So, if you lose SERVER1 for one reason or another, you actually lose 4 nodes and your majority… So even if your Primary was on SERVER2 when this happens, because it can’t reach anymore the majority of the nodes, it will step down and become secondary and you will be in the situation where you are at the moment: read only. Which means your cluster isn’t Highly Available because just one failing hard drive can bring everything down.

Also, SERVER1 has to share its hardware (RAM, CPU, SSD, Network) with the 4 mongods that you have running on this machine which isn’t ideal. It’s as good as one mongod really with access to 100% of the hardware.

Cheers,
Maxime.

2 Likes

That was my development server and I was in the midst of moving the 1P-2S-1A replica set from SERVER1 to SERVER2 since server1 is unstable.

My idea was to create replicas in server2 and then slowly disconnect the ones in server1 until everything is moved.

Not sure if that is the best method but it seemed safe for me. But I did not know the minimum 4 are needed. I read the documentation and my interpretation of “majority of the votes” was whoever got the most votes from the remaining voters. Guess I didnt understand it properly. Still I wonder why though?

Thanks I learnt something new today. Although I do not understand why this rule though.

A MongoDB Replica Set always needs to be able to reach the majority of the voting nodes to maintain a Primary. So if you have a 3 nodes RS - 2 need to be alive at all time. If you have 5 nodes, 3. 7 nodes => 4.

If that wasn’t the case, let’s imagine the following scenario:
SERVER 1: 4 nodes.
SERVER 2: 3 nodes.
So 7 nodes total. Let’s imagine I just need 3 to have a Primary instead of 4. Let’s now imagine that you have a network partition between SERVER 1 and 2. But they can still communicate with your driver.

On SERVER 1: 4 nodes can still see each other. They can elect a Primary.
On SERVER 2: 3 nodes can still see each other. Because we decided for the example that 3 was enough, they will ALSO elect a Primary… You now have your driver connecting to 2 Primaries and writing to one or the other randomly… Soon the content of your database doesn’t make any sense anymore as you are basically reading and writing randomly on what is now 2 separated and independent Replica Set.

This is called a split-brain issue and you NEVER EVER want that. This is one of the main reason why this rule exists. It will prevent you from ever having to deal with this situation.

In the “real” world on SERVER 2, because these 3 nodes can’t reach the majority of the nodes, they won’t be able to elect a Primary and you are safe to recover once the network partition is fixed. Meanwhile, normal activity can carry on on your SERVER1 with its 4 nodes (but that’s really still a bad idea :smiley: !).

I hope this helps :slight_smile: !

If you want to learn more, I really recommend that you have a look to this free training on MongoDB University which cover all these concepts.

https://university.mongodb.com/courses/M103/about

Cheers,
Maxime.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.