Every 2 to 5 minutes performance tanks

Hi,

About every 2-5 minutes the performance tanks and basically all I/O by mongod stops. In mongotop I can observe read times of up to 3 minutes. At some point it randomly snaps back to normal performance and dumps around 500 MB to disk.

The setup consists of two nodes, one primary and one replica. Hardware performance wise, there shouldnt be a problem. Normal data throughput is around 1.000 ops/s with mostly writes. Query targeting is below 100 and read ops below 1ms.

An example mongotop output during this “lagging” period looks like this:
ns total read write
db1.a 130317ms 0ms 130317ms
db1.b 43463ms 0ms 43463ms
db2.a 24959ms 19ms 24939ms
db3.a 16666ms 0ms 16666ms
db3.b 16171ms 0ms 16171ms
db4.a 13758ms 0ms 13758ms
db5.b 13731ms 0ms 13731ms
db6.a 13729ms 0ms 13729ms
db6.b 13720ms 0ms 13720ms
db6.c 13718ms 0ms 13718ms

What is pretty obvious is that total does not match read and write, but I havent found documentation on why that is though.

Would love to get some external input on this as I have not been able to figure out whats wrong with my installation.