I have been working with Kafka and Mongodb for a few months now. I am running into problems with the Kafka -> Mongodb sink connector, in that it cannot cope with the number of records I’m throwing at it. I’m processing around 100,000 records per second. I dont expect mongo to keep up with that but I’m only hitting about 1000-2000 records a second into mongo. I’m using upserting, using the primary key, any tips would be helpful. When we dump data straight in from SQL server we get much better through put, so its not server spec. I have increased batch size and max tasks, to no avail.
Thanks in advance.