Replication failures after initializing 2nd replication cluster

I created the 2nd cluster on ports 27004/27005/25006 and brought up the cluster and added them the the replication cluster.

I get these messages every so often:

2018-10-31T18:23:54.270+0000 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Faile
d to connect to 127.0.1.1:27006, in(checking socket for error after poll), reaso
n: Connection refused
2018-10-31T18:23:54.271+0000 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Faile
d to connect to 127.0.1.1:27005, in(checking socket for error after poll), reaso
n: Connection refused

MongoDB Enterprise m103-repl-2:PRIMARY> rs.status()
{
“set” : “m103-repl-2”,
“date” : ISODate(“2018-10-31T18:24:39.885Z”),
“myState” : 1,
“term” : NumberLong(1),
“syncingTo” : “”,
“syncSourceHost” : “”,
“syncSourceId” : -1,
“heartbeatIntervalMillis” : NumberLong(2000),
“optimes” : {
“lastCommittedOpTime” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“readConcernMajorityOpTime” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“appliedOpTime” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“durableOpTime” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
}
},
“members” : [
{
“_id” : 0,
“name” : “192.168.103.100:27004”,
“health” : 1,
“state” : 1,
“stateStr” : “PRIMARY”,
“uptime” : 963,
“optime” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“optimeDate” : ISODate(“2018-10-31T18:24:35Z”),
“syncingTo” : “”,
“syncSourceHost” : “”,
“syncSourceId” : -1,
“infoMessage” : “”,
“electionTime” : Timestamp(1541009434, 2),
“electionDate” : ISODate(“2018-10-31T18:10:34Z”),
“configVersion” : 3,
“self” : true,
“lastHeartbeatMessage” : “”
},
{
“_id” : 1,
“name” : “m103:27005”,
“health” : 1,
“state” : 2,
“stateStr” : “SECONDARY”,
“uptime” : 557,
“optime” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“optimeDurable” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“optimeDate” : ISODate(“2018-10-31T18:24:35Z”),
“optimeDurableDate” : ISODate(“2018-10-31T18:24:35Z”),
“lastHeartbeat” : ISODate(“2018-10-31T18:24:38.571Z”),
“lastHeartbeatRecv” : ISODate(“2018-10-31T18:24:39.077Z”
),
“pingMs” : NumberLong(0),
“lastHeartbeatMessage” : “”,
“syncingTo” : “192.168.103.100:27004”,
“syncSourceHost” : “192.168.103.100:27004”,
“syncSourceId” : 0,
“infoMessage” : “”,
“configVersion” : 3
},
{
“_id” : 2,
“name” : “m103:27006”,
“health” : 1,
“state” : 2,
“stateStr” : “SECONDARY”,
“uptime” : 549,
“optime” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“optimeDurable” : {
“ts” : Timestamp(1541010275, 1),
“t” : NumberLong(1)
},
“optimeDate” : ISODate(“2018-10-31T18:24:35Z”),
“optimeDurableDate” : ISODate(“2018-10-31T18:24:35Z”),
“lastHeartbeat” : ISODate(“2018-10-31T18:24:38.573Z”),
“lastHeartbeatRecv” : ISODate(“2018-10-31T18:24:38.714Z”
),
“pingMs” : NumberLong(0),
“lastHeartbeatMessage” : “”,
“syncingTo” : “m103:27005”,
“syncSourceHost” : “m103:27005”,
“syncSourceId” : 1,
“infoMessage” : “”,
“configVersion” : 3
}
],
“ok” : 1
}

I don’t get it, but after a while everything seemed to work.

The cluster itself looks fine.

As someone else indicated in another thread, this could possibly be related to running seven MongoDB instances on one VM, on underlying hardware which could be getting bogged down. I mean, I wouldn’t throw out that theory :slight_smile:

But that would be an easy way out… so perhaps it’s a good idea to actually investigate that idea, by checking the resources of your physical computer and doing a quick check on how the VM is doing resource-wise.