Mongo v4.4.3 crash "invariant in client_out_of_line_executor.cpp"

Hi, all
I have a problem with mongo 4.4.3, which crashes around 32.000 connections.
Arrived towards the 32k, mongo makes an error of assertion (on this piece of code) and this crashes the server.

ClientOutOfLineExecutor::~ClientOutOfLineExecutor() noexcept {
    invariant(_isShutdown);
}

Log :

{"t":{"$date":"2021-02-09T02:33:45.899+01:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn165377","msg":"client metadata","attr":{"remote":"XXXXXXX","client":"conn165377","doc":{"driver":{"name":"nodejs","version":"3.6.3"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"4.19.0-9-amd64"},"platform":"'Node.js v10.19.0, LE (legacy)"}}}
{"t":{"$date":"2021-02-09T02:33:45.907+01:00"},"s":"F",  "c":"-",        "id":23079,   "ctx":"listener","msg":"Invariant failure","attr":{"expr":"_isShutdown","file":"src/mongo/db/client_out_of_line_executor.cpp","line":58}}
{"t":{"$date":"2021-02-09T02:33:45.913+01:00"},"s":"F",  "c":"-",        "id":23080,   "ctx":"listener","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2021-02-09T02:33:45.913+01:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"listener","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2021-02-09T02:33:45.917+01:00"},"s":"E",  "c":"CONTROL",  "id":31430,   "ctx":"listener","msg":"Error collecting stack trace","attr":{"error":"unw_get_proc_name(559C30C01C41): unspecified (general) error\nerror: unw_step: unspecified (general) error\nunw_get_proc_name(559C30C01C41): unspecified (general) error\nerror: unw_step: unspecified (general) error\n"}}
{"t":{"$date":"2021-02-09T02:33:45.917+01:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"listener","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"559C30C01C41","b":"559C2DF13000","o":"2CEEC41"}],"processInfo":{"mongodbVersion":"4.4.3","gitVersion":"913d6b62acfbb344dde1b116f4161360acd8fd13","compiledModules":[],"uname":{"sysname":"Linux","release":"4.19.0-14-amd64","version":"#1 SMP Debian 4.19.171-2 (2021-01-30)","machine":"x86_64"},"somap":[{"b":"559C2DF13000","elfType":3,"buildId":"6C8A93F8D2B544901FC58C1CCD203AEA182627B5"}]}}}}
{"t":{"$date":"2021-02-09T02:33:45.917+01:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"listener","msg":"  Frame: {frame}","attr":{"frame":{"a":"559C30C01C41","b":"559C2DF13000","o":"2CEEC41"}}}

Has anyone ever had the problem ? Do you have any idea where it came from / how to solve it ?
Thanks

Hi all
We found the “issue”. Just forgot to increase the service limits with systemctl edit mongod.service

[Service]
LimitNOFILE=infinity  #<<< HERE
LimitNPROC=infinity  #<<< HERE

PermissionsStartOnly=true
ExecStartPre=/bin/sh -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
ExecStartPre=/bin/sh -c "echo never > /sys/kernel/mm/transparent_hugepage/defrag"

Mongo 4.3 gave us the message “Too many open files” while Mongo 4.4 crashed without this information.

Hope this can help someone in the future :slight_smile:

1 Like