Unable to reach PRIMARY after I dropped and changed bind IP to m103

When I was connected to primary (192.168.103.100:27001), I used rs.stepdown() which led to stepdown, then I removed it and added m103:27001, but it didn’t work, so I closed the shell, set the bind IP in all conf files to m103 and local host and now I am unable to reach primary or secondary as well.

mongod -f mongod-repl-1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 2121
child process started successfully, parent exiting
vagrant@m103:~$ mongo --host "m103-repl/m103:27001" -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin"
MongoDB shell version v3.6.16
connecting to: mongodb://m103:27001/?authSource=admin&gssapiServiceName=mongodb&replicaSet=m103-repl
2020-02-01T10:04:29.640+0000 I NETWORK  [thread1] Starting new replica set monitor for m103-repl/m103:27001
2020-02-01T10:04:29.642+0000 I NETWORK  [thread1] Successfully connected to m103:27001 (1 connections now open to m103:27001 with a 5 second timeout)
2020-02-01T10:04:29.643+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:29.643+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:29.643+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to 192.168.103.100:27001 (1 connections now open to 192.168.103.100:27001 with a 5 second timeout)
2020-02-01T10:04:29.644+0000 W NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Unable to reach primary for set m103-repl
2020-02-01T10:04:29.645+0000 I NETWORK  [thread1] Successfully connected to 192.168.103.100:27001 (1 connections now open to 192.168.103.100:27001 with a 0 second timeout)
2020-02-01T10:04:29.662+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:29.662+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:29.663+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:30.166+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:30.169+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:30.171+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:30.674+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:30.675+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:30.676+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:31.269+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:31.270+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:31.271+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:31.773+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:31.774+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:31.774+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:32.277+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:32.278+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:32.279+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:32.783+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:32.784+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:32.784+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:33.288+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:33.289+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:33.289+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:33.793+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:33.794+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:33.794+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:34.297+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:34.298+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:34.298+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:34.800+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:34.801+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:34.802+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:35.304+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:35.305+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:35.305+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:35.809+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:35.810+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:35.811+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:36.315+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:36.316+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:36.317+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:36.820+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:36.821+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:36.821+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:37.324+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:37.325+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:37.325+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:37.828+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:37.829+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:37.829+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:38.332+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:38.333+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:38.334+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:38.836+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:38.837+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:38.837+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:39.340+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:39.341+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:39.341+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:39.843+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:39.844+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:39.845+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:40.348+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:40.349+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:40.349+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:40.853+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:40.854+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:40.854+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:41.356+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:41.357+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:41.358+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:41.860+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:41.861+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:41.862+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:42.365+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:42.366+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:42.367+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:42.869+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:42.871+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:42.871+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:43.373+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:43.374+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:43.375+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:43.880+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:43.881+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:43.881+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:44.383+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:44.384+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:44.385+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:44.387+0000 E QUERY    [thread1] Error: Could not find host matching read preference { mode: "primary", tags: [ {} ] } for set m103-repl :
connect@src/mongo/shell/mongo.js:263:13
@(connect):1:6
exception: connect failed
vagrant@m103:~$ mongo --host "m103-repl/m103:27002" -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin"
MongoDB shell version v3.6.16
connecting to: mongodb://m103:27002/?authSource=admin&gssapiServiceName=mongodb&replicaSet=m103-repl
2020-02-01T10:04:52.453+0000 I NETWORK  [thread1] Starting new replica set monitor for m103-repl/m103:27002
2020-02-01T10:04:52.454+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:52.455+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:52.455+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
2020-02-01T10:04:52.957+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:52.958+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:52.958+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 2 checks in a row.
2020-02-01T10:04:53.461+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:53.461+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:53.462+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 3 checks in a row.
2020-02-01T10:04:53.963+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:53.963+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:53.963+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 4 checks in a row.
2020-02-01T10:04:54.469+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:54.469+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:54.469+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 5 checks in a row.
2020-02-01T10:04:54.971+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:54.971+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:54.971+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 6 checks in a row.
2020-02-01T10:04:55.472+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:55.473+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:55.473+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 7 checks in a row.
2020-02-01T10:04:55.974+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:55.975+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:55.975+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 8 checks in a row.
2020-02-01T10:04:56.476+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:56.476+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:56.476+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 9 checks in a row.
2020-02-01T10:04:56.977+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:56.978+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:56.978+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 10 checks in a row.
2020-02-01T10:04:57.479+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:57.479+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:57.479+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 11 checks in a row.
2020-02-01T10:04:57.981+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:57.982+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:58.484+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:58.484+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:59.066+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:59.066+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:04:59.568+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:04:59.569+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:00.070+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:00.070+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:00.573+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:00.573+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:01.075+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:01.075+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:01.576+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:01.577+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:02.079+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:02.080+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:02.581+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:02.581+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:02.581+0000 I NETWORK  [thread1] Cannot reach any nodes for set m103-repl. Please check network connectivity and the status of the set. This has happened for 21 checks in a row.
2020-02-01T10:05:03.088+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:03.088+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:03.590+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:03.590+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:04.097+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:04.099+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:04.602+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:04.602+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:05.106+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:05.106+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:05.607+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:05.608+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:06.110+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:06.110+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:06.612+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:06.613+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:07.115+0000 W NETWORK  [thread1] Failed to connect to 127.0.1.1:27002, in(checking socket for error after poll), reason: Connection refused
2020-02-01T10:05:07.115+0000 W NETWORK  [thread1] Unable to reach primary for set m103-repl
2020-02-01T10:05:07.116+0000 E QUERY    [thread1] Error: connect failed to replica set m103-repl/m103:27002 :
connect@src/mongo/shell/mongo.js:263:13
@(connect):1:6
exception: connect failed

Remove the name of the replica set and connect directly to the 27001 node:
mongo --host "m103:27001" -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin"

Once connected, check the status and conifg of the replica set:
use admin
rs.status()
rs.config()

Next, check which nodes are running:
ps -ef | grep [m]ongo

… the error message is telling you that (at some point) you added two nodes 127.0.0.1:27002 and 127.0.0.1:27003 on the m103-repl replica set that are not reachable. You need to remove those nodes and add it on the m103 or 192.168.103.100 hostname.

The first node is acting as secondary while other two are not reachable.
Should I remove them using rs.remove() and then again add both of them?

MongoDB Enterprise m103-repl:SECONDARY> rs.status()
{
        "set" : "m103-repl",
        "date" : ISODate("2020-02-02T09:05:18.851Z"),
        "myState" : 2,
        "term" : NumberLong(2),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(0, 0),
                        "t" : NumberLong(-1)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1580550589, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1580550589, 1),
                        "t" : NumberLong(2)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.103.100:27001",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 25,
                        "optime" : {
                                "ts" : Timestamp(1580550589, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2020-02-01T09:49:49Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 3,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 1,
                        "name" : "m103:27002",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2020-02-02T09:05:18.631Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Connection refused",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                },
                {
                        "_id" : 2,
                        "name" : "m103:27003",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2020-02-02T09:05:18.631Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Connection refused",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -1
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1580550589, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1580550589, 1),
                "signature" : {
                        "hash" : BinData(0,"whzcKEPRSJRxEkQZ9aLkBCJjV2I="),
                        "keyId" : NumberLong("6788369667309174785")
                }
        }
}

MongoDB Enterprise m103-repl:SECONDARY> rs.config()
{
        "_id" : "m103-repl",
        "version" : 3,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.103.100:27001",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "m103:27002",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "m103:27003",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5e35223db481ebce5ebaa03d")
        }
}

You missed a step :arrow_up:

vagrant 1962 1 10 09:30 ? 00:00:01 mongod -f mongod-repl-1.conf

What does that indicate?

when i ran this command as soon as I started the machine mongo --host "m103:27001" -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin" it said connnection failed therefore I initialized mongod for first node using mongod -f mongod-repl-1.conf

The point is, you started the first node, but what about the other two nodes?

Do I have to start them manually? Because that wasn’t the case before as when I started the first, other two were reachable and up automatically

Yes, you need to create config files for them and start them up.

Started all the nodes manually and now I am getting error of nodes not configured properly. Can you take a look?

vagrant@m103:~$ mongo --host "m103:27001" -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin"
MongoDB shell version v3.6.16
connecting to: mongodb://m103:27001/?authSource=admin&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("485d5d6a-ac81-48e2-9720-209e4c2bddfc") }
MongoDB server version: 3.6.16
Server has startup warnings:
2020-02-02T17:53:43.201+0000 I STORAGE  [initandlisten]
2020-02-02T17:53:43.201+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-02-02T17:53:43.201+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-02-02T17:53:44.311+0000 I CONTROL  [initandlisten]
2020-02-02T17:53:44.311+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2020-02-02T17:53:44.311+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2020-02-02T17:53:44.311+0000 I CONTROL  [initandlisten]
2020-02-02T17:53:44.311+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2020-02-02T17:53:44.311+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2020-02-02T17:53:44.311+0000 I CONTROL  [initandlisten]
MongoDB Enterprise m103-repl:OTHER> rs.status()
{
        "state" : 10,
        "stateStr" : "REMOVED",
        "uptime" : 64,
        "optime" : {
                "ts" : Timestamp(1580550589, 1),
                "t" : NumberLong(2)
        },
        "optimeDate" : ISODate("2020-02-01T09:49:49Z"),
        "lastHeartbeatMessage" : "",
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "infoMessage" : "",
        "ok" : 0,
        "errmsg" : "Our replica set config is invalid or we are not a member of it",
        "code" : 93,
        "codeName" : "InvalidReplicaSetConfig",
        "operationTime" : Timestamp(1580550589, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1580550589, 1),
                "signature" : {
                        "hash" : BinData(0,"whzcKEPRSJRxEkQZ9aLkBCJjV2I="),
                        "keyId" : NumberLong("6788369667309174785")
                }
        }
}
MongoDB Enterprise m103-repl:OTHER> rs.config()
{
        "_id" : "m103-repl",
        "version" : 4,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.103.100:27001",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "m103:27002",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "m103:27003",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 3,
                        "host" : "m103:27001",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5e35223db481ebce5ebaa03d")
        }
}
MongoDB Enterprise m103-repl:OTHER>

Have a look and you will notice that you now have 4 . members in the config of your replica set. If you know how to alter configs, remove the 192 member. If you’re not sure how, it’s best you build it from scratch… shutdown the nodes and delete all the data files.

Hi @Sachin_54317,

Can you please confirm that you were able to do the labs after starting from scratch ?

Please let me know if you are still stuck with this issue.

Thanks,
Shubham Ranjan
Curriculum Services Engineer