Mongodb pub/sub using Tailable Cursors in Nodejs with clusters

Working with Data Drivers
1

Hi everyone, like the title, I have found a bug in this package. It essentially uses mongodb tailable cursors to provide a message queue.

It works well in normal environments but when using it with clusters there seems to be some data leaks and some messages are not delivered as expected.

Does capped collections with tailable cursors have known bug when running with clusters?

Has anyone any idea about how this could be fixed?

Thanks

2

Welcome to the community @Daniel_Lando!

I’d suggest building on the MongoDB Change Streams API rather than directly using tailable cursors. Change Streams provide a supported API for resumable event streams from replica sets and sharded clusters.

Tailing the oplogs for a sharded cluster adds some complexity. For more details, see the two part series Tailing the MongoDB Oplog on Sharded Clusters and Pitfalls and Workarounds for Tailing the Oplog on a MongoDB Sharded Cluster. Note that these posts predate the Change Streams API, which was introduced in MongoDB 3.6 to provide a supported and scalable interface.

Regards,
Stennie

2 Likes
3

Hi @Stennie_X! Thanks so much for your answer. I will try to implement that solution using watch isntead of tailable cursors, will update you on the result

4

I think I’m missing something or maybe I haven’t explained the problem well.

When I try your solution I get this error:

The $changeStream stage is only supported on replica sets

I don’t use replica sets on my db, when I speak about clusters I mean nodejs clusters module (multiple instances of the same process using the same mongodb instance).

So back to my question in such environment is there the possiblity that some messages are not delivered using tailable cursors? If so, is there any other solution?

5

Hi Daniel,

Thanks for clarifying your use of “clusters”. I assumed in context that you were referring to a sharded cluster, since that is a more challenging environment for working with tailable cursors. Deployment type (standalone, replica set, or sharded cluster) and your version of MongoDB server would be definitely helpful details to include for future questions.

I’d still suggest using the Change Streams API, although the issue with your message queue is more likely related to concurrency than tailable cursors.

Can you provide more detail on how you are using the Node cluster? Do you have multiple workers watching and processing the same events?

Change streams require a replica set because data change events are based on the replica set oplog. However, you can enable this on a single node (if you don’t have any requirements for redundancy or failover) by converting a standalone to a single node replica set.

To do so, follow the tutorial to Convert a Standalone to a Replica Set without adding any additional members.

Regards,
Stennie

6

Hi @Stennie_X,

Thanks again, I have to inform you that I have fixed my problem, It was a bug in the package query used in the find method. On startup it was using something like:

var lastId = new mongodb.ObjectId()
collection.find({_id: {$gt: lastId}})

This was working as expeted in normal environment but with clusters seems this cause some data leaks. Fixed by creating a method to fetch last inserted document in the capped collection and using that id as lastId :slight_smile:

Thanks for your support, have a good day!

1 Like