com.mongodb.MongoNodeIsRecoveringException: Command failed with error 11600 (InterruptedAtShutdown)

Ops and Admin
#1

we have a 6 Shards Cluster with 3 routers and 1 config replica set, we started getting below error when connecting from Routers, couldnt find any specific issue related to the cluster, i can see cluster available up and running, and issue seems intermittent

com.mongodb.MongoNodeIsRecoveringException: Command failed with error 11600 (InterruptedAtShutdown): ‘interrupted at shutdown’ on server XXX:XXX. The full response is {​​​​​​​"ok": 0.0, “errmsg”: “interrupted at shutdown”, “code”: 11600, “codeName”: “InterruptedAtShutdown”, “operationTime”: {​​​​​​​"$timestamp": {​​​​​​​"t": 88608, “i”: 2}​​​​​​​}​​​​​​​, “$clusterTime”: {​​​​​​​"clusterTime": {​​​​​​​"$timestamp": {​​​​​​​"t": 88608, “i”: 2}​​​​​​​}​​​​​​​, “signature”: {​​​​​​​"hash": {​​​​​​​"$binary": {​​​​​​​"base64": “XXXXXXXXXXXX”, “subType”: “00”}​​​​​​​}​​​​​​​, “keyId”: XXXXXXXXXXXXXX}​​​​​​​}​​​​​​​}​​​​​​​ at com.mongodb.internal.connection.ProtocolHelper.createSpecialException(ProtocolHelper.java:242) ~[mongodb-driver-core-4.1.1.jar:na] at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:171) ~[mongodb-driver-core-4.1.1.jar:na] at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:359) ~[mongodb-driver-core-4.1.1.jar:na] at com.mongodb.internal.connection.InternalStreamConnection.receive(InternalStreamConnection.java:316) ~[mongodb-driver-core-4.1.1.jar:na] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:215) ~[mongodb-driver-core-4.1.1.jar:na] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144) ~[mongodb-driver-core-4.1.1.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

#2

Hi @Bhaskar_Avisha

First of all, welcome to MongoDB Community Forum… :slightly_smiling_face:

I believe error 11600 means that the client is trying to do an operation on a server that is shutting down.

So, if you provide the driver with a connection URI that specifies a replica set, the driver should reconnect to the new primary as soon as it’s available. See Connection String URI Format, specifically the replica set option section.

I hope it works for you.
Let us know if it or still persists.

In case of any doubts please feel free to reach out.

Regards,
Kushagra

#3

Thanks @Kushagra_Kesav

I am getting this error , when application is connecting through mongos , this is not a direct connection to replica set.

#8

Hi @Bhaskar_Avisha,

As far as I can tell, the com.mongodb.MongoNodeIsRecoveringException is a possible error that happens when you’re connected to a replica set. From the linked page:

An exception indicating that the server is a member of a replica set but is in recovery mode, and therefore refused to execute the operation. This can happen when a server is starting up and trying to join the replica set.

However I don’t think there’s enough information here to determine what’s going on. Could you post more details:

  1. What is your MongoDB version
  2. What is your Java driver version
  3. What is the connection string you used in your Java app (as mentioned by @Kushagra_Kesav)
  4. Typically, when does this error happen? Can you determine a pattern to it? For example, do you see this error when there’s a maintenance, network interruptions, high load, etc. ?
  5. How about if you try connecting to the mongos using some other method, e.g. the mongo shell, a driver from another language (Python, Node, etc.) do you still see the same error, or does it only happen with the Java driver?

Best regards,
Kevin

2 Likes