Kafka source connector - How can I watch for dynamic db names?

We have dynamic creation of database with fix prefix string. eg test_abc, test_xyz.

They have same collections inside. So I want to watch db names with regex or some other means.

How could I achieve this? I need to push every changes of collection to kafka topics in those dbs.

You should be able to use a pipieline to watch for your dynamic database names.

The Change Stream Output shows what these events look like. In your source connector define a pipeline

“pipeline”: “[{”$match": {“ns.db”:

{"regex\": /^(mydb1|mydb2)/}

}}]",

you can also use $in for explicit matches but since you dynamic, use regex

1 Like