Get the existing documents from selective collections from a database

Hi team,
I am using MongoDb Kafka Source connector to produce the existing document in Atlas DB into the Kafka topic. I want to exclude a collection from the database from which the documents are fetched.
I set the pipeline option as:

[{"$match": {“ns.coll”: {"$regex": /^(?!collection_to_be_excluded).*/}}}]

along with copy.existing set to true. But I am not seeing any existing records from the allowed collections.
Using pipeline works fine when not copying existing data. So the newly added records in the allowed collections will be produced into the Kafka topic.

I tried setting copy.existing.pipeline similar to pipeline value above, but still no record is getting into the topic.
How can I filter the collections for existing documents?

Many thanks.

cc: @Robert_Walters

Setting copy.existing.namespace.regex as:

^(database_to_watch.(?!collection_to_be_excluded$)).+$

will work. But is there a way we can make use of pipeline here?

Are you using the free tier or shared tier MongoDB Atlas size?

It is not working for both the free as well as dedicated (M30) tier.

Will the regex work for you or do you need to use a pipeline for copy.existing? If the latter can you describe the use case?

Never mind. The version 1.5.1 of the MongoDb connector fixed the issue.
This was the bug in previous version: https://jira.mongodb.org/browse/KAFKA-131

Thanks a lot for your help.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.