Hi, is there any convenient way to monitor lag/delay between oplog and kafka source connector? We got an incident of losing offset due to oplog shrunk window, so now we would like to monitor it.
What tool would you like to use to monitor ? Right now everything is logged through the kafka connect logs.
We using prometheus and grafana, but I’m not sure that there’s any way to get the metrics I want (oplog cursor vs source connector current offset)
The mongodb connector uses Change Streams not the oplog, are you wanting to monitor the delta between oplog and change stream and/or changestream and its message in a kafka topic?
Changestream last item and currently processed item by source connector, to be able to see if there’s any lag
Hi, did you figured out a way to monitor source connector?, any recommendations ?
We have a work item to provide a more formal way of monitoring. https://jira.mongodb.org/browse/KAFKA-229. I believe this captures your requirements, if it doesn’t please feel free to add to that ticket. There might be a way to query the timestamps between change streams and what is present in the kafka topic. I’ll do some research and repost.
@Robert_Walters , Thanks for the reply. This Jira item captures what i am looking for. Let us know if you find any simple solution to get the “source” lag.