Very likely this is a poorly written client application. Opening connections unnecessarily instead of using connection pools.
ss -tn "sport = :27017" or netstat -tn | grep :27017 will give you a list of mongo connections to this host, assuming default port, allowing you to target where the problematic connections are coming from.
How would I able to trace this out on my client. It is ultimately the mongo carrying the blame. Is there any way to create a dump on mongo to prove that client is misbehaving?
this error in mongo oplog when the closewait started increasing
2021-03-18T08:49:20.547+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host M04:27717 because the pool meets constraints; 2 connections to that host remain open
2021-03-18T08:49:36.601+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host M03:27717 because the pool meets constraints; 2 connections to that host remain open