Timeouts in mongos

Hello,

in our MongoDB Cluster (3 mongos, 3 data-replicas, 3 config-replicas) sometime application catch errors like this

2020-09-17 16:25:47,2547 [48] () ERROR
An unhandled exception has occurred: An exception occurred while opening a connection to the server.
MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. 
---> MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server.
---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
   --- End of stack trace from previous location where exception was thrown ---
   at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytesAsync(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.IsMasterHelper.GetResultAsync(IConnection connection, CommandWireProtocol`1 isMasterProtocol, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.ConnectionInitializer.InitializeConnectionAsync(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at AspectCore.DynamicProxy.AspectActivator.InvokeTask[TResult](AspectActivatorContext activatorContext)
   at Mango.Platform.ProductInventory.Controllers.v2.ProductController.GetProductsInternalAsync(ProductFilteredRequest request) in e:\\b\\Agent2\\13\\s\\Mango.Platform.ProductInventory\\Controllers\\v2\\ProductController.cs:line 342
   at Mango.Platform.ProductInventory.Controllers.v2.ProductController.GetProductsPost(ProductFilteredRequest request) in e:\\b\\Agent2\\13\\s\\Mango.Platform.ProductInventory\\Controllers\\v2\\ProductController.cs:line 104
   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware.Invoke(HttpContext httpContext)
   at Mango.Platform.Diagnostics.Middlewares.RequestResponseLoggingMiddleware.Invoke(HttpContext context)
   at ServiceRegistry.AspCore.Middleware.MartiExecutionContextMiddleware.InvokeAsync(HttpContext context)
   at ServiceRegistry.AspCore.Middleware.ServerStatisticsMiddleware.InvokeAsync(HttpContext context)
   at Mango.Platform.Diagnostics.Middlewares.ExceptionHandlingMiddleware.Invoke(HttpContext context)

at this time in mongos-log (log level=3)
many timeouts like this

2020-09-17T16:25:47.094+0300 D NETWORK [TaskExecutorPool-0] Timer received error: CallbackCanceled: Callback was canceled
(complete logs attached).

When we connect directly to data-replicas - we have no prolem like this, problem is not reproduced.

So we need to investigate - what is the reason os this timeouts, when we working with mongos?

Application and mongos working on the same Windows-servers (Windows 2012 R2).

Data and config replicas working on others Linux-servers (4.1.12-124.32.1.el7uek.x86_64).
MongoDB version 4.0.12.

I’ve asked before in https://jira.mongodb.org/browse/SERVER-50942, but they recommended to ask in this community…

Please, help to find a reason of this timeouts…