I am looking to shutdown and bring back up my kubernetes cluster to save money. How do I properly shutdown MongoDB in a kubernetes environment to support this?
I would suggest a normal shutdown of the host / k8s should work in most circumstances. Most sequences would involve send a kill SIGTERM to running containers, initiating a shutdown of mongod.
With journalling any write should be persisted before being ack’d to the client.
I frequently tear down and start replicasets in docker without even considering this. They always start back up flawlessly.
Could you elaborate what happens under the hood when k8s issues SIGTERM?
When you call shutdown on a primary, doesn’t it start primary reelection process? If so, it is a vicious circle, because potential candidates (secondaries) are also being shut down after k8s SIGTERM.
Do you know it or do you just suggest it - because I can say from experience, that even with Mongodb operator, there is always this error “Detected unclean shutdown” .
So either
a) we must delete the lock file
b) or there must be a proper shutdown procedure avail (which still does not exist)
I don’t know (for now) how you would issue a command to remove the container itself in this case, but to clean shutdown mongod instances, you need to connect to the admin database on each instance and run db.shutdownServer().
There should be some scripting around for this purpose. have you tried to search for such?
PS: check my answer in your other post about how to one-line with mongo shell.
Continue here