Lab - Initiate a Replica Set Locally - Failed to connect to m103-repl/192.168.103.100:0, in(connect), reason: No such file or directory

Hi, I think I followed all the instructions but I am getting the following error every 30 seconds.

2019-03-26T02:42:10.934+0000 W NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to m103-repl/192.168.103.100:0, in(connect), reason: No such file or directory
  

The output for rs.status() is as follow:

MongoDB Enterprise m103-repl:PRIMARY> rs.status()
{
        "set" : "m103-repl",
        "date" : ISODate("2019-03-26T02:51:28.482Z"),
        "myState" : 1,
        "term" : NumberLong(5),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1553568685, 1),
                        "t" : NumberLong(5)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1553568685, 1),
                        "t" : NumberLong(5)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1553568685, 1),
                        "t" : NumberLong(5)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1553568685, 1),
                        "t" : NumberLong(5)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.103.100:27001",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 3371,
                        "optime" : {
                                "ts" : Timestamp(1553568685, 1),
                                "t" : NumberLong(5)
                        },
                        "optimeDate" : ISODate("2019-03-26T02:51:25Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1553566813, 1),
                        "electionDate" : ISODate("2019-03-26T02:20:13Z"),
                        "configVersion" : 3,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 1,
                        "name" : "192.168.103.100:27002",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 2333,
                        "optime" : {
                                "ts" : Timestamp(1553568685, 1),
                                "t" : NumberLong(5)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1553568685, 1),
                                "t" : NumberLong(5)
                        },
                        "optimeDate" : ISODate("2019-03-26T02:51:25Z"),
                        "optimeDurableDate" : ISODate("2019-03-26T02:51:25Z"),
                        "lastHeartbeat" : ISODate("2019-03-26T02:51:27.664Z"),
                        "lastHeartbeatRecv" : ISODate("2019-03-26T02:51:28.195Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "192.168.103.100:27001",
                        "syncSourceHost" : "192.168.103.100:27001",
                        "syncSourceId" : 0,
                        "infoMessage" : "",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "m103-repl/192.168.103.100",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2019-03-26T02:51:27.300Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "HostUnreachable",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1553568685, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1553568685, 1),
                "signature" : {
                        "hash" : BinData(0,"Dxvg4KxaaT3INvP08NsSpZZuEKc="),
                        "keyId" : NumberLong("6672512452053172226")
                }
        }
}
  

When I run the script “validate_lab_initialize_local_replica_set” I get the following:
Replica set ‘m103-repl’ not configured correctly - make sure there is one primary and two secondaries, and all three nodes are healthy.

Any tips on debugging this? I think i messed up when starting the 3rd node.
How do I stop the replica set, remove the 3rd member and do rs.add correctly?

I think 3rd node is added with wrong port(0)
You can use rs.remove

Please check this link

Removing a Member from Replica Set or Replica Set as Whole

1 Like

Hi Ramachandra,

Thank you!

The following is for my reference:

  1. Kill pid for the third/anomalous mongod process.
  2. Remove it from rs using rs.remove()
  3. Initiate the node using mongod -f /etc/mongod-repl-3.conf
  4. rs.add the correct node and check rs.status()