All application instances receiving events from change stream. How can we loadbalance events amongst the instances?

All application instances receiving events from change stream. How can we loadbalance events amongst the instances?

There is many ways to do it. You must do it yourself. Here are some ideas, some more applicable than others depending of your use-cases.

  1. Have different processes listing to different events or data sources (collections/databases).
  2. Have a central handler that dispatches to many listeners. Any message queues system can help you with the dispatch.

Hello Steeve,

Regarding point 1, Since its just two instance of same code base running on two replicas I am really confused how can we really point make different instance to listen to different events.

Second suggestion will entail introduction of shared cache (a new point of failure), which will be fed by both instances with duplicate data, so there will be a need to handle duplicates.

Whats the industry standard here? How is this feature used in production by other developers?

I want to emphasis the point:

For example, if your use-case relates to phone numbers. You have one server listening to phone numbers that ends with an even number and the other one that ends with odd numbers. So only one type of event is sent to one and the other type sent to the other. So you have the same code base handling 2 unique sets of event. But you might also want one handler listens to inserts and an other one listen to updates. You might do both in the same code base right now. But logically it is 2 code bases into the same server and you do an if to dispatch to one of the 2 internal code bases.

Yes. But most message queue systems handle that. A simple event cache can even be a MongoDB replica set that is fed by your multiple listeners with upserts which kind of make sure there is not duplicate.

Kafka seems to have some momentum. RabbitMQ is also popular.