What if only one SECONDARY node left?

Hi all,
Let’s assume I have a replica set of 3 members: 1 PRIMARY and 2 SECONDARY nodes. Let’s assume 2 of them went down, so I am left with only one SECONDARY node which will never become a PRIMARY because of the majority rule, hence no read/write operations will be available for the Client App. Isn’t this a big issue? If I bring up a replica set of 3 members, I would expect at least one of them to be receiving and giving back data. What should I do in this situation?
Thanks.

1 Like

If a majority of members isn’t available, then we can’t guarantee durability of any new write operations, so it seems that MongoDB errs on the side of caution, guaranteeing durability of your writes by not allowing any writes while the replica set is in a state which can’t guarantee durability of your writes.

(reads will be allowed as long as the client specifies a read preference which allows reading from a secondary node)

Have you read about my misadventure where I got into a similar situation?

In this situation, we have two courses of action, depending on whether the nodes which have gone down are temporarily or permanently down.

If the broken nodes are fixable, fix them and bring them back up, and once a majority are up and running, they’ll elect a primary automatically and normal service will be resumed.

If the broken nodes aren’t fixable (e.g. the servers’ CPUs have burned out), find another couple of servers from somewhere, start a mongod instance on each one, and then add them to the replica set, and remove the two dead mongod instances from the replica set. Once you’ve done this, you’ll have a replica set full of healthy nodes, and they’ll have elected a primary before you know it

This second approach is complicated by the fact that you can’t normally do write operations (including administrative commands such as adding and removing nodes to/from a replica set) on a secondary node, which isn’t great when the only node you’ve got for managing the replica set is a secondary, but we’re saved by the fact that the rs.reconfigure() method allows you to force execution on a secondary node.

I’m guessing you’ve probably just started chapter 2? If so, then by the time you’ve completed chapter 2 and read my other post, all of this should make sense.

Of course, as always, I’m just a fellow student with limited experience, so contributions from people with more experience than me are most welcome :slight_smile:

2 Likes

If only the secondary node was left - I would think you should still be able to read… NO writes.

1 Like

Thank you for your replies guys! Special thanks to Simon for sharing similar problem and a solution to it!

1 Like