Read operations flock to Secondary node

Currently, 23 hours out of 24 hours are normally read as any Secondary, but 1 hour are concentrated in a certain secondary.

What kind of problem is this happening?

And I wonder if there were any other cases like this, and if there were, what was the cause at that time.

Hi @Kim_Hakseon,

This does not sound like a familiar problem.

Can you share some more details:

  • What are you using to measure connections concentrated on a single secondary?

  • What specific MongoDB driver or client version is the affected application using?

  • What is your version of MongoDB server?

  • What does your connection string look like (with any hostnames or user credentials redacted)?

  • Is there a pattern for the hour of the day when the secondary reads appear to target a specific server?

  • Does the observed effect appear to have a common duration (full hour, half hour, …)?

With a specific time of day correlation my first suspicion would be to look for a scheduled job (eg backup or batch process) which might be targeting one secondary.

Regards,
Stennie

I can explain in detail, but i havnot received yet accurate information.
also due to security issues, it is difficult to disclose the details.
please appreciate your understanding.

=
What are you using to measure connections concentrated on a single secondary?
→ I don’t know but they are using 3rd monitoring solution.
The connection of MongoDB is checking in real time through the monitoring solution.

During rush hour, other secondary nodes maintain a normal level of connection, and a specific secondary node has connections about x20 times more than other nodes.

What specific MongoDB driver or client version is the affected application using?
→ java mongodb driver version 2.14

What is your version of MongoDB server?
→ MongoDB 3.4.x

when rush hour, there is no regular pattern other than what happens.

What does your connection string look like (with any hostnames or user credentials redacted)?
→ I don’t know for this contents.

Is there a pattern for the hour of the day when the secondary reads appear to target a specific server?
→ There is no regular pattern that except in when the issue occurs during rush hour.

Does the observed effect appear to have a common duration. (full hour, half hour, …)?
→ Because an issue occurred and the service was affected, the secondary node was excluded from the replica set grop. so I don’t know exactly this contents.

With a specific time of day correlation my first suspicion would be to look for a scheduled job (eg backup or batch process) which might be targeting one secondary.
→ I was received contents that there was no such thing, but I will check again.

=
When analyzing issue,

  1. What should I check in the log?
  2. Would i be helpful that check settings value in the mongodb options?

Additionally, may I know what these logs mean?

User Assertion: 11600:interrupted at shutdown src/mongo/db/operation_context.cpp 159

SocketException: remote: {IP} error: 9001 socket exception [CLOSED] server {IP}

These phrases are constantly popping up, but I don’t know what they mean, and I don’t know what to check.