Due to the nature of my data (being very many collections) I have a lot of problems starting mongod. I have about 140k *.wt files on disk, although during the normal running of the server I only have about 2k open pointers as once. So this large number of files is only a problem on startup, which takes about 5 mins.
I’ve posted about ulimits before and have resolved that problem in systemd by using a drop-in config that sets LimitNOFILE=200000 plus the corresponding nproc config as mentioned here. I think this side of things is resolved.
However I have hit a new ceiling. Today (amidst upgrade to 4.4), starting mongod causes the server to run out of memory. The box has 4G of RAM and a 2G swap disk. It only runs MongoDB and under normal operation the mongod process seems to use about half the available RAM.
So I suppose my question is what can I do to get the server to start with my 4G of RAM? Do I need a bigger box just to get it started? If so, is it possible to calculate the ram I’ll need based the number of files? Perhaps increasing the size of the swap disk would fix it? Disk space is not a problem.
The next size up of my Linode VPS is 8G and will double the monthly cost for all my mongodb servers. Not the end of the world, but as I only seem to need this RAM on startup, I’m wondering if I can hold off on that upgrade.
I tried increasing my swap partition to 4GB and still got memory exhaustion. I don’t understand memory management in Linux well enough to know if I did this correctly. It appeared to me that the swap space wasn’t being used, but I’m out of my depth here.
The only way I’ve been able to fix this is to trash the data directory and start from empty. It takes MUCH longer to start, but doesn’t suffer the same problems.
Final update until I have to go through all this again. I realised that my system swappiness was 1 as per the recommendations, so I issued sysctl vm.swappiness=50 and the server started without exhausting memory. I don’t know if this is a coincidence, because supposedly the value of 1 swaps “only to avoid out-of-memory problems” which should have meant it would work.
Thanks for the reply. Had that article been around 8 years ago, we wouldn’t be having this conversation. Likewise if WiredTiger had been the storage engine. However I am dealing with what I’ve got until such time as I have the capacity to completely redesign my system.
I don’t know how to answer your question accurately, but I usually have about 1G of spare RAM when the system is running. At any one time only about 2k file pointers are open.
I think you are fine with a swappiness at 50 when mongod is starting. But then your linux might be more tempted to use the swap if you are starting some more RAM intensive activities. I guess what you could do is reduce the swappiness once mongod is started so the system is less tempted to use the swap once the startup is done.
Also, make sure to monitor your swap usage at this point. If your mongod starts using the swap after the startup - I guess you definitely need more RAM then.
What I’m about to say is really not accurate and should not be taking for an absolute truth, but usually I would recommend to have about 10-20% of the node size in RAM.
So for example if you have a 200GB replica set (200GB per node). I would have around 20 to 40GB of RAM. Unless there is really something out of the ordinary going on, this should leave you plenty of RAM for the indexes which should represent about 10-20% of the RAM and the rest is for mongod to keep frequently queried documents in RAM and resolve queries.
This ratio didn’t fall completely from the sky though. It’s my feeling, but it’s also more or less the ratio you get by default when you take a cluster on MongoDB Atlas.
