Performance degradation after upgrading to 4.4

Our MongoDB deployment looks like this: 6x mongos instances (n1-highcpu-8), 3x config instances (n1-standard-1), 15 (shards) *3 (replica members) mongod instances (n1-highmem-16). Hosted on a GCP cloud. After upgrading to 4.4, we noticed immediate performance degradation across the entire system. Even queries by sharding key, which returns one document from the small collection (~30k documents), started to degrade in performance. For example, we saw cases where log entry on the mongos side says that the Slow query took 6-7 sec, but we have nothing on the mongod side which means, that query took less than 100ms. We tried to find bottlenecks, tried to temporarily resize instances, add more mongos instances, but the results were the same. Then the 4.4.1 version came out, but the upgrade didn’t change anything. So we decided to downgrade to the 4.2 release. At this time, we monitored various system parts as downgrade was performed. And the performance was back as soon as we restarted mongos instances with the 4.2 binaries and stayed at the same levels as we downgraded all the shards one by one.
An interesting thing, that basic server metrics like CPU, Load, Memory, Disk Activity didn’t change at all during upgrade and downgrade. Just mongos instances became slower with version 4.4 for some reason.
Any ideas what could slow down mongos instances so much? May others experienced something like this?

Is this on MongoDB Atlas or are you running your own cluster on MongoDB?

It is our own cluster with a community version. OS Ubuntu 18.04 installed by extracting binaries from tgz.

I spoke to our engineering team. They recommend raising a SERVER ticket on our JIRA system.

Hi Žygimantas Stauga and welcome to the new community forums! I’d very much like to learn more about your performance issue – please let me know if you have any issues filing a SERVER project ticket.

Thanks
Dan

2 Likes

Hi, sorry for the late response. Just did it: https://jira.mongodb.org/browse/SERVER-51104

1 Like

Thanks! We’ll follow up on the ticket.