Failed to mlock: errno:12 Cannot allocate memory

Hi Team,

Our mongo services are restarting continuously and below are the error log details:

OS and mongo version details:

mongo 3.2.22 and CentOS release 6.9 (Final)

2020-10-17T11:34:43.114-0500 F -        [conn1310] Failed to mlock: errno:12 Cannot allocate memory
2020-10-17T11:34:43.114-0500 I -        [conn1310] Fatal Assertion 28832
2020-10-17T11:34:43.114-0500 I -        [conn1310]

***aborting after fassert() failure

2020-10-17T11:34:43.119-0500 F -        [conn1310] Got signal: 6 (Aborted).

 0xc8b852 0xc8a789 0xc8af92 0x33f740f7e0 0x33f7032495 0x33f7033c75 0xc0d722 0x6b5cec 0x770dc5 0x771025 0x771146 0x7be5a3 0x7bfaff 0x79647b 0x7b23d7 0x7b4338 0xbbbc08 0xbbc8dd 0xbcba4d 0xbbb286 0x69c3a5 0xc36405 0x33f7407aa1 0x33f70e8bcd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"88B852","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"88A789"},{"b":"400000","o":"88AF92"},{"b":"33F7400000","o":"F7E0"},{"b":"33F7000000","o":"32495","s":"gsignal"},{"b":"33F7000000","o":"33C75","s":"abort"},{"b":"400000","o":"80D722","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"2B5CEC","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"400000","o":"370DC5","s":"_ZN5mongo5scram15generateSecretsERKNS_9SHA1BlockE"},{"b":"400000","o":"371025","s":"_ZN5mongo5scram15generateSecretsERKNS0_15SCRAMPresecretsE"},{"b":"400000","o":"371146","s":"_ZN5mongo5scram19generateCredentialsERKSsi"},{"b":"400000","o":"3BE5A3","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorISsSaISsEEPSs"},{"b":"400000","o":"3BFAFF","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPSs"},{"b":"400000","o":"39647B","s":"_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPSs"},{"b":"400000","o":"3B23D7"},{"b":"400000","o":"3B4338"},{"b":"400000","o":"7BBC08","s":"_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"400000","o":"7BC8DD","s":"_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi"},{"b":"400000","o":"7CBA4D","s":"_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE"},{"b":"400000","o":"7BB286","s":"_ZN5mongo7Request7processEPNS_16OperationContextEi"},{"b":"400000","o":"29C3A5","s":"_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"400000","o":"836405","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"33F7400000","o":"7AA1"},{"b":"33F7000000","o":"E8BCD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.22", "gitVersion" : "105acca0d443f9a47c1a5bd608fd7133840a58dd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.23.1.el6.x86_64", "version" : "#1 SMP Thu Sep 26 12:05:41 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "9FDE7E083B126FA0B29530BD37E03D51927CDF1B" }, { "b" : "7FFDBFCB5000", "elfType" : 3, "buildId" : "147632A13846D848909EB52353F6102CF932E1E5" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "5A37D12297A649A37081AB63AC6B520444079986" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "0435315E0E6DC8BCF08DE1794B2A84D431DE835A" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "FDF3A36FFFE08375456D59DA959EAB2FC30B6186" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1F7E85410384392BC51FA7324961719A10125F31" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "8A852AC42F0B64F0F30C760EBBCFA3FE4A228F12" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9350579A4970FA47F3144AD8F40B183B0954497D" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "85104ECFE42C606B31C2D0D0D2E5DACD3286A341" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "814F2290D172521A3FD8581389E3E78A4A182379" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1CC2165E019D43F71FDE0A47AF9F4C8EB5E51963" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F0BE1166EDCFFB2422B940D601A1BBD89352D80F" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc8b852]
 mongos(+0x88A789) [0xc8a789]
 mongos(+0x88AF92) [0xc8af92]
 libpthread.so.0(+0xF7E0) [0x33f740f7e0]
 libc.so.6(gsignal+0x35) [0x33f7032495]
 libc.so.6(abort+0x175) [0x33f7033c75]
 mongos(_ZN5mongo13fassertFailedEi+0x82) [0xc0d722]
 mongos(_ZN5mongo24secure_allocator_details8allocateEmm+0x57C) [0x6b5cec]
 mongos(_ZN5mongo5scram15generateSecretsERKNS_9SHA1BlockE+0x75) [0x770dc5]
 mongos(_ZN5mongo5scram15generateSecretsERKNS0_15SCRAMPresecretsE+0x25) [0x771025]
 mongos(_ZN5mongo5scram19generateCredentialsERKSsi+0x106) [0x771146]
 mongos(_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorISsSaISsEEPSs+0x1673) [0x7be5a3]
 mongos(_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPSs+0x34F) [0x7bfaff]
 mongos(_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPSs+0x2B) [0x79647b]
 mongos(+0x3B23D7) [0x7b23d7]
 mongos(+0x3B4338) [0x7b4338]
 mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x6C8) [0xbbbc08]
 mongos(_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x2ED) [0xbbc8dd]
 mongos(_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE+0x19D) [0xbcba4d]
 mongos(_ZN5mongo7Request7processEPNS_16OperationContextEi+0x876) [0xbbb286]
 mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x65) [0x69c3a5]
 mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0xc36405]
 libpthread.so.0(+0x7AA1) [0x33f7407aa1]
 libc.so.6(clone+0x6D) [0x33f70e8bcd]
-----  END BACKTRACE  -----

Please do the need ful.

Thanks
Raj

Hi @Raj_Sandiri,

This issue usually manifest as your ULIMITs OS setting is not configured according to our best practices:
https://docs.mongodb.com/v3.2/reference/ulimit/

Specifically for this error its

limit memlock unlimited unlimited

But I recommend checking all of them for all members.

Moreover 3.2 is not supported or developed anymore and its better to go to 3.6+

Best
Pavel