What’s the best way to sync mongodb data to elastic? I’m thinking that my service should publish message to a kafka queue and a worker will fetch it and update to elastic db. But after I research, there’s mongo-connector and logstash. What’s the difference if i use a message queue and what things should i be aware if i use mongo-connector/logstash? Thanks
Edit: someone told me that using kafka is more stable than mongo connector and logstash. After i read somemore theres also mongo kafka connector. Whats the difference between setting up my own queue and consumer vs using mongo kafka connector? Thanks
Kafka Connect is a native Kafka service that is built for integrating heterogenous data like MongoDB within Kafka. You could write your own consumer but you’ll have to deal with initial loading of data and keeping track of topic offsets to protect yourself in case of errors. In the end why bother coding all that, just use the service that is already designed for this purpose. The MongoDB Connector for Apache Kafka is the official connector supporting Kafka Connect.
In your scenario, simply use the MongoDB connector as a source and point to your MongoDB cluster. Then use the ElasticSearch Sink connector to pull data from the kafka topic to Elastic.
1 Like