Too many open files - Mongo 4.2

Hi Folks,

We have a MongoDB replica set configured - primary, secondary & arbiter. In the past few weeks one of the instances has crashed multiple times. The logs show the following :

2020-08-28T12:14:20.570+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.578+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.612+0000 I  SHARDING [conn54569] Marking collection *****.componenttypes as collection version: <unsharded>
2020-08-28T12:14:20.616+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.623+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.623+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.625+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.629+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.630+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files

The /lib/systemd/system/mongod.service file looks like this:

[Unit]
Description=An object/document-oriented database
Documentation=man:mongod(1)

[Service]
# Other directives omitted
# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (open files)
LimitNOFILE=64000
# (processes/threads)
LimitNPROC=64000
User=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongod.conf

[Install]
WantedBy=multi-user.target

The proc/pid/limits file is :

    Limit                     Soft Limit           Hard Limit           Units
    Max cpu time              unlimited            unlimited            seconds
    Max file size             unlimited            unlimited            bytes
    Max data size             unlimited            unlimited            bytes
    Max stack size            8388608              unlimited            bytes
    Max core file size        0                    unlimited            bytes
    Max resident set          unlimited            unlimited            bytes
    Max processes             64000                64000                processes
    Max open files            64000                64000                files
    Max locked memory         65536                65536                bytes
    Max address space         unlimited            unlimited            bytes
    Max file locks            unlimited            unlimited            locks
    Max pending signals       64122                64122                signals
    Max msgqueue size         819200               819200               bytes
    Max nice priority         0                    0
    Max realtime priority     0                    0
    Max realtime timeout      unlimited            unlimited            us 

Any ideas what could be causing this?

Mongo version 4.2.8
Ubuntu 16.04

Thanks in advance.

You are hitting the operating system limit for the number of open file descriptors which is non unusual on database servers. Please refer to your operating system documentation for how to increase this limit.

you can check it, from unix doing an ulimit
$ ulimit -a ( and set the appropiate limit )

@Willy_Latorre Setting with the ulimit command only effects the shell session. This parameter need to be set in the appropriate script(or init system) that starts mongod.

ulimit is referenced in the Operations Checklist
Specifically in UNIX ulimit Settings

1 Like

Hi Guys,

Thanks for all the responses. I had already set the Mongo recommended ulimit settings in the service file and the proc limits file showed that the 64000 was set. However when I ran ulimit -u as the Mongo user, it was returning 1024. I increased this limit to 100000, but the instance has crashed again.

Steps to increase limit to 100000

  1. Add fs.file-max = 100000 to /etc/sysctl.conf
  2. Refresh with new config - sudo sysctl -p
  3. Edit the /etc/security/limits.conf file & add following lines to it

'* soft nofile 100000
'* hard nofile 100000
mongodb soft nofile 100000
mongodb hard nofile 100000
root soft nofile 100000
root hard nofile 100000

  1. Add ‘session required pam_limits.so’ to /etc/pam.d/common-session
  2. Logout and login and ‘ulimit -a’ as Mongo user returns 100000

Any other ideas?

Cheers,
Niamh

You missed the critical piece:

This means systemd, upstart or init script. There are examples toward the end of UNIX ulimit Settings for upstart and systemd.

1 Like

After modifying the service script, did you run systemctl daemon-reload?

1 Like

Hey Chris,

Thanks for checking again, but I already have that in place in my /lib/systemd/system/mongod.service file…see original post.

I must be missing something else?

Thanks,
Niamh

Yep. See @ken.chen comment.

This might need to be updated in the documentation too.

Hey Ken / Chris,

The config in the service file has been in place for a long time, I inherited the system like this Although I recently upgraded this Mongo instance and issued systemctl daemon-reload at the Disable Transparent Huge Pages (THP) step, where I created the /etc/systemd/system/disable-transparent-huge-pages.service file.

Any other suggestions as to what I could be missing? I’m new to Mongo and your help is really appreciated.

Thanks,
Niamh