Hi !
I have a set up which includes three machines, m1, m2 and m3. These machines talks to each other using hostnames host-m1, host-m2 and host-m3. The IP to host mapping is retained in the /etc/hosts of each machine. For various reasons, I cannot employ a DNS server here.
So, the /etc/hosts of each machine looks something like this,
================== 10.46.51.1 =================
10.46.51.3 host-m3
10.46.51.2 host-m2
10.46.51.1 host-m1
================== 10.46.51.2 =================
10.46.51.3 host-m3
10.46.51.2 host-m2
10.46.51.1 host-m1
================== 10.46.51.3 =================
10.46.51.3 host-m3
10.46.51.2 host-m2
10.46.51.1 host-m1
Now, for various reasons, the hostnames sometime have to be changed. For example, this,
10.46.51.3 host-m3
10.46.51.2 host-m2
10.46.51.1 host-m1
might need to be changed to,
10.46.51.3 host-m3
10.46.51.2 host-m1
10.46.51.1 host-m2
on all the machines.
While this change is in progress, the mappings on all machines might not be the same. Meaning, on m1 “host-m3” maps to m3, while on m2 “host-m3” might map to m1
While this change is happening, I will sometime do a replica set init. When I do this, I observe the following,
- Init command goes through without any errors.
- The instance on which the replica set init was performed says it is in the “SECONDARY” state, while the other two members are in “STARTUP”.
- The member in “SECONDARY” state continues to receive heartbeats from the other two members.
- Connecting to members in “STARTUP” state and fetching status of the replica set give, “NotYetInitialized”
- Reads and Writes fails.
- Setup continued to stay in this state for well over 30 mins. It does not correct itself when the mappings get in sync.
- The setup corrects itself and reaches Primary-Secondary status if one of the instance in “STARTUP” state is restarted.
Can someone help me understand what is mongo’s recommended approach while init-ing a replica set in a scenario where the host to IP mappings on the all the members might not be in sync.
Additional details:
-
The Actual hostname to IP mappings on the machines where,
================== 10.46.51.5 =================
10.46.51.5 cvm-5
10.46.51.2 cvm-1
10.46.51.1 cvm-2
================== 10.46.51.1 =================
10.46.51.5 cvm-2
10.46.51.2 cvm-5
10.46.51.1 cvm-1
================== 10.46.51.2 =================
10.46.51.5 cvm-1
10.46.51.2 cvm-2
10.46.51.1 cvm-5 -
Mongodb Version 2.4.6 was running on centos 7