Connection refused error when one node of the replica set is down

I am getting the following error when one of the secondaries of my replica set is down to do the lab “Write with Failovers”. Can someone tell me how to correct this? The constant error is making the lab difficult to do. Thanks…

2019-06-30T16:38:25.301+0000 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 192.168.103.100:27002, in(checking socket for error after poll), reason: Connection refused

Here is my /etc/hosts file

127.0.0.1 m103.mongodb.university m103
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
192.168.103.100 m103 m103

what do your config files look like? Have you tried restarting vagrant? I had some errors and that is what I needed to do.

Here are the 3 configs. Node 1 is primary. Node 2 is the one that is down. Same error occurred when I dropped 3. I’ll try restarting vagrant and report the results.

Node 1 config:
storage:
dbPath: “/var/mongodb/db/1/”
systemLog:
destination: file
path: “/var/mongodb/db/mongod1.log”
logAppend: true
operationProfiling:
mode: slowOp
slowOpThresholdMs: 50
net:
bindIp: “localhost,192.168.103.100”
port: 27001
security:
authorization: enabled
keyFile: /var/mongodb/pki/m103-keyfile
processManagement:
fork: true
replication:
replSetName: m103-repl

Node 2 config
storage:
dbPath: “/var/mongodb/db/2/”
systemLog:
destination: file
path: “/var/mongodb/db/mongod2.log”
logAppend: true
operationProfiling:
mode: slowOp
slowOpThresholdMs: 50
net:
bindIp: “localhost,192.168.103.100”
port: 27002
security:
authorization: enabled
keyFile: /var/mongodb/pki/m103-keyfile
processManagement:
fork: true
replication:
replSetName: m103-repl

Node 3 config
storage:
dbPath: “/var/mongodb/db/3/”
systemLog:
destination: file
path: “/var/mongodb/db/mongod3.log”
logAppend: true
operationProfiling:
mode: slowOp
slowOpThresholdMs: 50
net:
bindIp: “localhost,192.168.103.100”
port: 27003
security:
authorization: enabled
keyFile: /var/mongodb/pki/m103-keyfile
processManagement:
fork: true
replication:
replSetName: m103-repl

Do you have a mongod listening on port 27002? Check the mongods which are running like this:

vagrant@m103:~$ ps -ef | grep mongo
vagrant   2056     1  1 12:12 ?        00:05:12 mongod -f /shared/lab3.2/csrs1.conf
vagrant   2144     1  1 12:12 ?        00:05:20 mongod -f /shared/lab3.2/csrs2.conf
vagrant   2233     1  1 12:12 ?        00:06:20 mongod -f /shared/lab3.2/csrs3.conf
vagrant   2410     1  3 12:13 ?        00:11:09 mongod -f /shared/lab3.2/node1.conf
vagrant   2504     1  3 12:13 ?        00:12:00 mongod -f /shared/lab3.2/node2.conf
vagrant   2611     1  3 12:13 ?        00:11:44 mongod -f /shared/lab3.2/node3.conf
vagrant   2725     1  2 12:14 ?        00:06:29 mongod -f /shared/lab3.2/node4.conf
vagrant   2826     1  2 12:14 ?        00:06:55 mongod -f /shared/lab3.2/node5.conf
vagrant   2935     1  2 12:14 ?        00:06:29 mongod -f /shared/lab3.2/node6.conf
vagrant   3089     1  0 12:14 ?        00:00:59 mongos -f /shared/lab3.2/mongos.conf
vagrant   3284  1927  0 12:17 pts/0    00:00:02 mongo --port 26000 -u m103-admin -p xxxxxxxxx
vagrant   4316  4280  0 12:56 pts/4    00:00:01 mongo --port 27001 -u m103-admin -p xxxxxxxxx
vagrant   4334  4281  0 12:57 pts/2    00:00:02 mongo --port 27005 -u m103-admin -p xxxxxxxxx
vagrant   8077  3381  0 17:35 pts/1    00:00:00 grep --color=auto mongo
vagrant@m103:~$

That’s just the output when I run that command, the results will be different for you, but it’ll tell you the configuration files that your mongod instances are using. Look at each of those config files to identify the one which is being used on port 27002, and then look for the error in that config file.

There is no mongod running on port 27002 because I shut it down as per the lab instructions. The configs are already part of this thread.

Sorry, I didn’t read the question correctly. So you’ve shut down the mongod listening on port 27002 in order to observer the behaviour of the replica set when one of its nodes is out of action?

Where are you seeing the error message

? In a log file, or when trying to connect the mongo shell to the replica set? If you’re trying to connect the shell to port 27002 then that won’t work because you’ve stopped the mongod which was listening on that port. What happens if you try to connect the shell to a port that one of the other mongods in the replica set is listening on?

1 Like

if you try to connect like this it should work:
mongo --host “m103-repl/192.168.103.100:27002” -u m103-admin -p m103-pass

if you try to connect like this it will not:
mongo --host “192.168.103.100:27002” -u m103-admin -p m103-pass

you have to tell it that the connection is part of a cluster - which is wha the m103-repl/ does…

1 Like