Kafka source connector and copy.existing on shared-tier instances

Working with Data Connectors & Integrations
1

Hi there,
I’ve encountered strange behavior of Kafka source connector when I creating new connector with copy.existing setting I’m getting only 101 documents out of 12k+ while I’m using M5 cluster, but after upgrade to M10, and reproducing the same flow it just works and I’m getting all the documents.

So, I would like to ask if there may be a bug in the connector so it is not failing/crashing/logging any error but instead just says INFO Finished copying existing data from the collection(s). (com.MongoDB.kafka.connect.source.MongoSourceTask:553)

2

Hi Vlad,

Currently with shared/free tier clusters in Atlas, there is some proxy work that happens under the covers which causes this behavior with the Kafka Connector. We have a ticket to address this, however for now, if you’d like to copy.existing you will need to use a dedicated cluster M10+.

Thanks,
Rob

1 Like
3

Thanks for clarification, any chance that I can get a link to the ticket to watch it?

4

https://jira.mongodb.org/projects/KAFKA/issues/KAFKA-201

This should be addressed in the next release - 1.5.

2 Likes