Multiple writes on same document within journal commit interval are not updated in the document

Working with Data
#1

with journaling enabled, if multiple write operations on the same document occur with in the CommitInterval , only the last received write operation is persisted and writes before that are ignored/missed.

According to docs, every write operation will have a record with unique identifier in the journal. So even if there are write operations on the same document, all the writes should be persisted without fail. But in my case only last received write operation is only performed.

#2

Hi @Perumalla_Giridhar, welcome to the community.

By commit interval, do you mean the setting storage.journal.commitIntervalMs?

If yes, then the journal in question is WiredTiger’s internal journaling mechanism (write ahead log), which I don’t believe is accessible from MongoDB. WiredTiger uses this journal mechanism to guard against acknowledged write loss in case the server was killed unexpectedly.

How did you determine that some writes are being ignored by WiredTiger? Could you elaborate on the testing method?

Best regards,
Kevin

#3

Hi Kevin,
Yes, I meant storage.journal.commitIntervalMs

I saw from my application logs that if multiple patch requests are sent at almost same time and on same document, I get status ok (200) from mongodb but only last sent patch request will take effect on the document.
For patch requests sent with some time gap (little more than 100 milli seconds), all patch requests will take effect.
My write concern is having default value (i.e., {w : 1}).
Later I tried changing my write concern to {w: 1, j: true}. From then all patch requests, even when sent at same time, on same document, took effect.
My question here is, how come j:true solved the problem?
j:true is only related to acknowledgement. It confirms on-disk entry of patch requests which is only useful if mongodb crashes.
Why are in-memory records in case of default {w: 1} acknowledgement setting is causing issue even when there is no mongodb restart or failure?
Is there any problem with my understanding? Please correct me if I’m wrong anywhere.

#4

Hi @Perumalla_Giridhar,

Sorry I’m a bit confused. Are you using MongoDB via a driver, a REST interface, or some other method? I ask because MongoDB doesn’t reply in HTTP status code, so OK 200 doesn’t really exist in MongoDB lingo.

Could you post the actual command you send to MongoDB (or the code you use to interface with MongoDB), and why you think it’s not the expected outcome? Please also post your MongoDB version and your driver version if applicable.

From what I understand, you’re looking for read-your-own-writes capability. Is this correct? If yes, then you might benefit from causal consistency. See read your own writes and Causal Consistency and Read and Write Concerns for more detailed explanations.

Best regards,
Kevin

1 Like
#5

Hi Kevin

Let me explain you the whole story.

I have multiple versions of an application running which are clients of mongodb, say app.v0 and app.v1
these apps communicate to mongodb through a python eve interface. where the application send http request to the interface and get response 200 if successful.

If a patch request is made to interface then that interface will send the necessary update command to mongodb. To know more about details of interface, please refer python eve . the default write concern which python eve uses is {w: 1}

Now app.v0 and app.v1 process documents one after other. most of the cases they process same document at same time. The processing time is also almost same most of the cases. after processing they update their answer by adding a new field in the document. app.v0 add {v0: 'itsanswer'} to the document and app.v1 will add {v1: 'itsanswer'} field to the document. if both the update request take effect then my final state of document will have the {answers: {v0: 'itsanswer', v1: 'itsanswer'}}

since app.v0 and app.v1 send patch request at same time, and with default write concern {w: 1} only the last send patch request is been successful. In this case my document will have either {answers: {v0: 'itsanswer'}} or {answers: {v1: 'itsanswer'}}. In app.v0 and app.v1 , v0 is the production version. so if production answer is missed i get errors in later stage of pipeline.

Later when i changed the write concern settings of python eve (interface) to use {w: 1, j: true} the problem was solved surprisingly.

Now can you understand my question in the previous comment? according to my understanding j: true would only help in case of mongodb crash or restart. Is that correct?

my mongodb version is 4.2.7 . I have followed the exact procedure as mentioned in the official documentation during its installation.

#6

Hi @Perumalla_Giridhar

I believe what you’re seeing is a race condition between the two eve apps. I don’t think it has anything to do with MongoDB or eve at all.

What I think happened is:

  1. app.v0 tries to update one document
  2. At the same time, app.v1 tries to update the same document
  3. Both apps gets the “pre” version of the document, and accordingly set fields v0 and v1 respectively
  4. Both updates are done using something like replaceOne, where it replaces the whole document

In this scenario, it is a race between v0 and v1. The winner basically has their version of the document. This is why you see only field v0 sometimes, or only field v1 some other times.

However, setting j: true has the effect of throttling both apps, meaning:

  1. app.v0 updates one document, and did so. Field v0 is now in the document
  2. app.v1comes after v0’s update, and thus sets v1. Both fields are now in the document

However, the throttle effect cannot be guaranteed to happen since it was not explicitly designed, so I think occasionally you’ll still see either v0 or v1 like in the non-throttled case.

What I don’t know is whether eve is designed to work like this. That is, multiple eve instances interfacing with a single database instance. In a simplified term, I’m not sure if multiple eve processes are “thread-safe”.

If eve was never designed with these scenario in mind, then you would have to implement an external signalling process so that multiple instances of eve can work together. Alternatively, you might be able to raise an issue in eve github repository.

Of course, this is all assuming that you have parallel eve interfaces running at the same time. If, however, you have a single eve instance interfacing with a single MongoDB instance, and are calling eve’s endpoint from two different REST clients, then we need to dig deeper into what could possibly cause eve to have this race.

Best regards,
Kevin

#7

Hi Kevin,

I am using single interface instance (python eve running in a separate container ). all the clients make their patch requests to this container only. But i am running python eve endpoint under uwsgi in that container. Do you think race condition can occur because of uwsgi ?

#8

Hi @Perumalla_Giridhar

Unfortunately I don’t know enough about eve nor uwsgi to provide you with an answer. What I do know is that the condition was unlikely to be caused by MongoDB, and the symptoms thus far seem to show a race condition.

I think you can start the investigation by using a change stream on that collection (note that this feature requires a replica set). Using change stream, you can monitor all the modifications in that collection, and verify what MongoDB was doing to that collection.

Another avenue worth pursuing is to elevate the log level or elevate the profiling level. By default, MongoDB records queries taking more than 100ms. You can set this number to e.g. -1ms to ensure that it captures all queries into the log, and trace what happened from the server side.

Note that elevated log levels or profiling level would have performance drawbacks, so please be cautious with regard to the workload, and turn them back to their original settings once the investigation is done.

Best regards,
Kevin