Seeing an abundance of the below in a sharded cluster environment hosted in AWS. Any insight as to how to debug? Have tinkered with tcp keepalive on the servers (currently set to 120) and maxIdleTime on the client without any noticeable change.
MongoDB Server Version: 4.4.2
Java Driver: 'org.mongodb:mongodb-driver-reactivestreams:1.13.1
‘io.reactivex.rxjava3:rxjava:3.0.3’
Architecture: arm64
"stack_trace":"java.util.concurrent.ExecutionException: com.mongodb.MongoSocketReadException: Exception receiving message
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at com.creativeradicals.openio.pipeline.persist.feed.FeedMultiSaver.accept(FeedMultiSaver.java:107)
at com.creativeradicals.openio.pipeline.persist.feed.FeedMultiSaver.accept(FeedMultiSaver.java:34)
at com.creativeradicals.openio.rabbit.base.RequestConsumer.handleDelivery(RequestConsumer.java:42)
at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:104)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.mongodb.MongoSocketReadException: Exception receiving message
at com.mongodb.internal.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:569)
at com.mongodb.internal.connection.InternalStreamConnection.access$1200(InternalStreamConnection.java:76)
at com.mongodb.internal.connection.InternalStreamConnection$5.failed(InternalStreamConnection.java:520)
at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.failed(AsynchronousChannelStream.java:235)
at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.failed(AsynchronousChannelStream.java:203)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Unknown Source)
at java.base/sun.nio.ch.Invoker$2.run(Unknown Source)
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source)
... 3 common frames omitted
Caused by: java.io.IOException: Connection reset
at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishRead(Unknown Source)
at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(Unknown Source)
at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(Unknown Source)
at java.base/sun.nio.ch.EPollPort$EventHandlerTask.run(Unknown Source)
... 1 common frames omitted"}
Our application uses docker, and it looks like the version of Java we are on (OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.2+12) has patched the bug mentioned in that thread. Is there any other way to debug these constant connection reset errors and socket exceptions?
There’s no straightforward way to determine the root cause of connection reset errors. It’s not typically a driver bug that causes it. Rather, it’s either something happening in the MongoDB server or in the network between driver and server. I would look first at MongoDB server logs to see if there are any clues there. It’s possible that the server itself is closing the connection for some reason. If not, you’ll need to involve an expert in network administration, perhaps to employ a tool like Wireshark to figure out what’s happening, assuming that you can reproduce the error.
One other thought: if you’re able to test outside of Docker, that would be one way to rule that Docker itself as a contributing factor.
Our cluster is running on ARM64 CentOS machines in AWS. I see that the compatibility specs don’t list CentOS under ARM64. Is it worth trying to switch the underlying Operating System to Ubuntu?
.
Any more ideas? These client disconnects are happening frequently. As i stated we’ve followed the administrative guide to a tee. This is very disruptive to our application.
It does not sound like a MongoDB problem, but rather, a network, hardware, or hypervisor problem with net connectivity. In 2017 I saw something like this and it turned out to be net connectivity between 2 subnets in separate wings of a factory installation. Have your network people looked at this?
I am facing a similar issue here, while inserting the document or listing the database names from MongoDB server I am facing the exact above issue. Attaching logs below:
317 [main] INFO org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
682 [cluster-ClusterId{value=‘643f9d4e2e7bc45d1876b7a6’, description=‘null’}-10.7.202.205:27017] INFO org.mongodb.driver.cluster - Exception in monitor thread while connecting to server 10.7.202.205:27017
com.mongodb.MongoSocketReadException: Exception receiving message
at com.mongodb.internal.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:543)
at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:428)
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:289)
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255)
at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:83)
at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:33)
at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:106)
at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:63)
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:127)
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.SocketException: Connection reset
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
at com.mongodb.internal.connection.SocketStream.read(SocketStream.java:89)
at com.mongodb.internal.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:554)
at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:425)
… 9 more