Memory limit on K8S pod is not honored for WiredTigerCacheSize calculation

Ops and Admin
,
1

I am deploying mongo db 4.4.1 (windows servercore 1809 image) container on K8S. The memory limit on the ‘pod’ is set at 300M. However the container start fail with out of memory exception. Looking deeper in the logs, it turns out that wired tiger is trying build a cache of size ~3GB which is resulting in the exception.

Is this a known issue or am I missing something here?

relevant log lines -

{“t”:{“$date”:“2020-11-02T12:49:55.088+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}
{“t”:{“$date”:“2020-11-02T12:50:01.066+00:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:“2020-11-02T12:50:01.067+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648602, “ctx”:“main”,“msg”:“Implicit TCP FastOpen in use.”}
{“t”:{“$date”:“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:{“pid”:6496,“port”:27017,“dbPath”:“C:/PluginDataFolder”,“architecture”:“64-bit”,“host”:“documents-0”}}
{“t”:{“$date”:“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23398, “ctx”:“initandlisten”,“msg”:“Target operating system minimum version”,“attr”:{“targetMinOS”:“Windows 7/Windows Server 2008 R2”}}
{“t”:{“$date”:“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:{“buildInfo”:{“version”:“4.4.1”,“gitVersion”:“ad91a93a5a31e175f5cbf8c69561e788bbc55ce1”,“modules”:,“allocator”:“tcmalloc”,“environment”:{“distmod”:“windows”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}
{“t”:{“$date”:“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:{“os”:{“name”:“Microsoft Windows Server 2019”,“version”:“10.0 (build 17763)”}}}
{“t”:{“$date”:“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:{“options”:{“net”:{“bindIp”:“*”},“security”:{“authorization”:“enabled”},“storage”:{“dbPath”:“C:\PluginDataFolder”}}}}
{“t”:{“$date”:“2020-11-02T12:50:01.089+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=3071M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}
{“t”:{“$date”:“2020-11-02T12:50:01.093+00:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:12,“message”:“[1604321401:93157][6496:140717780783712], wiredtiger_open: __wt_calloc, 52: memory allocation of 32237280 bytes failed: Not enough space”}}

2

A memory limit is not the same as available memory(at least in docker, not sure about k8s, but I assume it is the same).

To limit mongod wiredtiger cache size, specifically set it in the configuration or via command line arguments.

If you run mongod in a container (e.g. lxc , cgroups , Docker, etc.) that does not have access to all of the RAM available in a system, you must set storage.wiredTiger.engineConfig.cacheSizeGB to a value less than the amount of RAM available in the container. The exact amount depends on the other processes running in the container. See memLimitMB .

1 Like
3

Hi @Hemant_Jain,

While the documentation note pointed out by Chris is a solid workaround, it does predate an improvement to detection of the memory constraint within a container/cgroup versus the total system memory: https://jira.mongodb.org/browse/SERVER-16571 (fixed in 3.6.13+, 4.0.9+, and 4.2+).

However, looking into this change further it appears to have been made specifically for Linux environments and would not help with a deployment running Windows in a container.

Can you please raise a new improvement issue in the SERVER project in the MongoDB JIRA issue tracker: http://jira.mongodb.org/browse/SERVER?

I can raise an issue on your behalf, but I expect the triage team may need further information on your deployment environment so it would be best for you to report directly.

Thanks,
Stennie

2 Likes
4

Thanks @Stennie_X and @chris for the suggestion.
I shall raise the issue as indicated.

5

@Hemant_Jain
Could you please confirm - are you starting MongoDB using the Mongodb Enterprise Operator? Or just the mongodb container without an Operator?
If it’s the Operator then the ubuntu is used as a based image for Database images and there should be no issues with wired tiger cache calculation.
Also as a small “nit” - the wired tiger memory shown in the logs is 32 MB, not GB.

1 Like
6

@Anton_Lisovenko I am not using the enterprise operator rather just the mongodb container.

Also as a small “nit” - the wired tiger memory shown in the logs is 32 MB, not GB.

Yes, you are right about the failed memory allocation, it failed while allocating a chunk of 32M. But my concern is about the ‘cache size’ wired tiger has decided to build (see the part in bold)

{“t”:{“$date”:“2020-11-02T12:50:01.089+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,

cache_size=3071M

,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}

1 Like
7

Raised the bug in Jira - https://jira.mongodb.org/browse/SERVER-52596

1 Like