MongoDB.live, free & fully virtual. Register Now MongoDB.live, free & fully virtual. Register Now

Application lag - delay in response time

Hello Everyone,

Please need your help on below issue. Application getting delay in response time when connecting to database or may be driver issue. Database using 3.2 community version and not sure on client driver side.

There is no slow queries has been captured and enough resources are available. No iowait and No reads/writes wait.

I am unable to debug the issue whether its from client driver or the database. I was able to pull up the information on java driver connections to mongodb and below is the output.

On tier: globalbroker_ncw
On node: globalbroker-7978d69ddd-g26n7–1
Start time: 11/18/20 11:06:10 AM
Snapshot GUID: 1cf5834d-fea0-45f1-8007-0c325388be42

com.mongodb.client.internal.MongoCollectionImpl:executeFindOneAndReplace:711 (method time = 0 ms, total time = 24612 ms)
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor:execute:193 (method time = 0 ms, total time = 24612 ms)
  com.mongodb.operation.BaseFindAndModifyOperation:execute:69 (method time = 0 ms, total time = 24612 ms)
   com.mongodb.operation.CommandOperationHelper:executeRetryableCommand:445 (method time = 0 ms, total time = 24612 ms)
    com.mongodb.operation.OperationHelper:withReleasableConnection:424 (method time = 0 ms, total time = 24612 ms)
     com.mongodb.operation.CommandOperationHelper$4:call:452 (method time = 0 ms, total time = 24612 ms)
      com.mongodb.internal.connection.DefaultServerConnection:command:123 (method time = 0 ms, total time = 24612 ms)
       com.mongodb.internal.connection.DefaultServerConnection:command:131 (method time = 0 ms, total time = 24612 ms)
        com.mongodb.internal.connection.DefaultServerConnection:executeProtocol:269 (method time = 0 ms, total time = 24612 ms)
         com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor:execute:200 (method time = 0 ms, total time = 24612 ms)
          com.mongodb.internal.connection.CommandProtocolImpl:execute:72 (method time = 0 ms, total time = 24612 ms)
           com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection:sendAndReceive:444 (method time = 0 ms, total time = 24612 ms)
            com.mongodb.internal.connection.UsageTrackingInternalConnection:sendAndReceive:99 (method time = 0 ms, total time = 24612 ms)
             com.mongodb.internal.connection.InternalStreamConnection:sendAndReceive:244 (method time = 0 ms, total time = 24612 ms)
              com.mongodb.internal.connection.CommandMessage:encode:57 (method time = 0 ms, total time = 154 ms)
               com.mongodb.internal.connection.RequestMessage:encode:138 (method time = 0 ms, total time = 154 ms)
                com.mongodb.internal.connection.CommandMessage:encodeMessageBodyWithMetadata:161 (method time = 0 ms, total time = 154 ms)
                 com.mongodb.internal.connection.RequestMessage:addDocument:188 (method time = 0 ms, total time = 154 ms)
                  com.mongodb.internal.connection.RequestMessage:addDocument:238 (method time = 154 ms, total time = 154 ms)
              com.mongodb.internal.connection.LoggingCommandEventSender:sendStartedEvent:78 (method time = 0 ms, total time = 24305 ms)
               ***com.mongodb.internal.connection.ProtocolHelper:sendCommandStartedEvent:281 (method time = 24051 ms, total time = 24305 ms)**
                com.mongodb.internal.connection.ByteBufBsonDocument:toJson:100 (method time = 0 ms, total time = 254 ms)
                 com.mongodb.internal.connection.ByteBufBsonDocument:toJson:110 (method time = 152 ms, total time = 254 ms)
                  com.mongodb.internal.connection.CompositeByteBuf:get:111 (method time = 0 ms, total time = 51 ms)
                   com.mongodb.internal.connection.CompositeByteBuf:get:116 (method time = 51 ms, total time = 51 ms)
                  com.mongodb.internal.connection.CompositeByteBuf:get:125 (method time = 0 ms, total time = 51 ms)
                   com.mongodb.internal.connection.CompositeByteBuf:get:130 (method time = 0 ms, total time = 51 ms)
                    com.mongodb.internal.connection.CompositeByteBuf:get:156 (method time = 51 ms, total time = 51 ms)
              com.mongodb.internal.connection.InternalStreamConnection:receiveCommandMessageResponse:289 (method time = 0 ms, total time = 153 ms)
               com.mongodb.internal.connection.InternalStreamConnection:receiveMessage:425 (method time = 0 ms, total time = 103 ms)
                com.mongodb.internal.connection.InternalStreamConnection:receiveResponseBuffers:554 (method time = 0 ms, total time = 103 ms)
                 com.mongodb.internal.connection.SocketStream:read:89 (method time = 103 ms, total time = 103 ms)
               com.mongodb.internal.connection.LoggingCommandEventSender:sendSucceededEvent:143 (method time = 0 ms, total time = 50 ms)
                com.mongodb.internal.connection.ProtocolHelper:sendCommandSucceededEvent:294 (method time = 50 ms, total time = 50 ms)
                 Custom(125ms): {"findAndModify":"BrokerState","query":{"_id":?},"fields":{},"sort":{},"new":?,"upsert":?,"update":{"_id":?,"serviceabilityObject":{"evo":?,"userRole":?,"level1":?,"level2":?,"level3":?,"dbRoot":?,"bo...