When you've added the same mongod to a replica set twice

I got into a right pickle whilst working on the lab “remove and re-add a node”. I’ve solved it now and completed the lab, but I’m posting my mistake and how I fixed it here because I’m sure other people are going to make the same mistake.

So, I’ve got a replica set with 3 nodes, listening using their IP address and port numbers 27001, 27002 and 27003. I need to remove one of them and re-add it using its hostname instead of IP address. I chose the one listening on port 27003 as the one to remove and re-add. I removed it OK, but when re-adding it, I made a typo and actually added the one listening on port 27002. Immediate flood of error messages, and I couldn’t execute any commands in the shell.

It was obvious what I’d done wrong, because the mongod instance listening on port 27002 was now trying to be two nodes of the replica set at once, which it clearly didn’t like. And I knew what I had to do - remove the node listening on m103:27002. But how, when I couldn’t execute any shell commands, and I needed to be in the shell in order to update the relevant documents in the local database (using the correct admin commands of course)?

I tried passing the rs.remove command to the mongo shell as an –eval command line argument, but that didn’t work, I got the same connection errors.

Killing and restarting the mongod instances didn’t help (not sure why I thought it would).

Deleting all the replica set’s files and re-creating it from scratch would probably have worked, but I wanted to find a better way before resorting to such desparate measures.

What I did in the end was this - connect to the one node that I hadn’t messed up, the one listening on port 27001, rather than the replica set

mongo --host "192.168.103.100:27001" -u etc

No errors this time. At least I could now do things like rs.status(), which informed me that two of my nodes, the ones which were actually the same mongod instance, were in a “not reachable/healthy” state, and the remaining one, listening on port 27001, was now a secondary node, presumably because it couldn’t contact a majority of the nodes in the replica set.

So, could I now remove the node that I added in error? No.

{
        "ok" : 0,
        "errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the \"force\" argument to override",
[removed the rest for brevity]
}

So I should only be running this command on the primary node, but I’m not going to have a primary node until I’ve removed that erroneous node. I’m willing to take a chance and force it to run on this secondary node. But there’s a slight problem, the rs.remove() command doesn’t appear to have a force argument. So maybe I can do it by editing the config document and passing the updated document to rs.reconfig(), which does have a force argument?

Get the config document…

var c = rs.conf()

Remove the last element from the members array (this is the node I added in error and want to remove) using the pop() method…

c.members.pop()

Check what the config document looks like now to ensure that I’ve removed the correct element of the array…

c

And seeing that the document looks as I’d expect, force a reconfiguration of the replica set…

rs.reconfigure(c, {force: true})

That returned an “ok”: 1, so check the state of my replica set…

rs.status()

… which tells me that I now have 2 nodes, one of which has stepped up to be primary, the other is a secondary, and neither appears to be complaining about being unhealthy. Phew, I seem to be back on track…

So I quit out of the shell, because I’m still connected to a single node rather than to the replica set, and start another shell connecting to the replica set, and this time I’m able to connect and re-add the node using its hostname rather than its IP address (being very careful this time to type the port number correctly). I ran the validation script, which gave me a code, which was accepted as the correct answer.

One thing bugging me though, the shell is now printing this warning every 30 seconds:

W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 127.0.1.1:27003, in(checking socket for error after poll), reason: Connection refused

My replica set appears to be working, as I can run admin commands, and I can also do things like show dbs, show collections and db.collection.findOne(), so why would I be getting this warning?

I hope I haven’t given too much away here about how to complete what would have been a fairly simple lab if I hadn’t made that silly typo, but please feel free to redact bits if I have.

Simon

Actually, ignore my last question about the warning from the shell, I found the answer here:

I edited my /etc/hosts file, changing the 127.0.1.1 entry to 127.0.0.1, waited 30 seconds, and instead of printing another warning, the shell printed

I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to m103:27003 (1 connections now open to m103:27003 with a 5 second timeout)

And it’s stopped printing messages now. So it was nothing to do with my typo and the actions I took to recover the situation.

2 Likes