Insert-heavy load performance questions

Hey there. I’ve performed a couple of simple benchmarks to see how Mongo behaves when inserting documents with j:true and w:1. The environments I used were my local machine with 8 core CPU and SSD, and Azure hosted VM with 4 core CPU and dedicated 256 GB SSD disk. I use the C# driver with shared singleton Mongo Client. The documents I insert are fairly small ones - no more than 2kb. The number of writers I tested are 1-5, 100 locally and distributed tests with up to 600 concurrent writers on Azure.
There are a couple of questions I’d like to clarify:

  1. When I increase the number of writers, Mongo is able to insert mode documents per second, as seen using mongostat. However, the IOPS do not increase linearly with the number of documents. For example, 200 documents correspond to 200 IOPS, 400 to 400, but inserting 500, 1k or 2k documents per second results in 500 IOPS consumed. Furthermore, the IOPS used are only a fraction of total available for disk: Azure disk can handle up to 2k IOPS and my local one is capable of thousands as well. Is there any internal buffering Mongo performs in order to write the entire batch of documents in one operation? I would expect the number of documents to be equal (or less than) to IOPS since j:true implies the document is flushed right away.
  2. Why does the number of inserted documents not grow linearly when increasing the number of writers? There’s plenty of CPU, RAM and IOPS left and the only locks I observed are IX ones for the db and collection.
  3. The most curious one: when I start a second mongod on the same machine and use basic hash to rotate between writing to them, IOPS increases and the number of written documents increases as well. When I use one mongod and multiple dbs/collections, nothing changes. Basically, somehow two mongod processes are able to provide better performance with the same resources dedicated. Is there anything which would cause this? I’d prefer not to start a new mongod for each 10 writers.

Further notes: the local file system is NTFS and it’s a Win 10 machine, the Azure one is the latest mongo docker image with the XFS filesystem on the dedicated Az disk.

Hi Vlad,

I’m not a performance expert nor very knowledgeable in Windows/Azure, but I’ll try to answer what I can.

MongoDB doesn’t actively try to buffer anything, so the plateauing IOPS number you’re seeing seem to imply that there is a bottleneck somewhere else (e.g. potentially not on your disk performance). Perhaps your singleton MongoClient is hitting a performance ceiling somewhere? Have you tried inserting with multiple client applications, or using other driver e.g. Java, node, etc.?

Another hint that seem to point toward unknown bottleneck is the fact that running two mongod and alternate write between them seem to push your IOPS usage higher, meaning the bottleneck was bypassed. Could it be the docker image you’re using is artificially limiting resources somehow? Have you tried using the Windows native mongod server and see similar behaviour?

Best regards,
Kevin

Hi,

MongoDB doesn’t actively try to buffer anything

I observe that Mongo consumes the same IOPS with X and X*3 number of events inserted per second (according to mongostat). Given that I believe IOPS are measured correctly, it does look as if Mongo buffers things.

Perhaps your singleton MongoClient is hitting a performance ceiling somewhere?

I don’t think so. The client also utilises multiple connections (usually roughly the same as the number of writers), so it’s not the connection re-use issue.

Have you tried inserting with multiple client applications, or using other driver e.g. Java, node, etc.?

I’ve tried multiple client applications up to 3 nodes. They behave the same as one node and offer the same throughput.

Could it be the docker image you’re using is artificially limiting resources somehow?

There are restrictions placed, but the resources utilised are nowhere near the limit.

Have you tried using the Windows native mongod server and see similar behaviour?

Yes, the behaviour is the same.

I’ve profiled the mongod process and it looks like the issue is the sub-optimal locking, which was fixed in later versions: https://jira.mongodb.org/browse/SERVER-43417

I tried mongo 4.4-rc and observed an increase in throughput up to x3 with the same setup and load patterns.