How can change stream update operations come with null fullDocument (when ChangeStreamFullDocumentOption.UpdateLookup was used)?

I’m getting a null fullDocument on a change stream update event from the C# Mongo Change Streams API when ChangeStreamFullDocumentOption.UpdateLookup was set in the options. What are the ways in which this can happen? Can it happen if someone updates the object and sets it to an empty or null document?

Example code:

    var options = new ChangeStreamOptions();
            options.FullDocument = ChangeStreamFullDocumentOption.UpdateLookup;

    using (var cursor = this.Database.Watch(options))
            {
                while (cursor.MoveNext())
                {
                    if (!cursor.Current.Any())
                    {
                        break;
                    }

                    using (var enumerator = cursor.Current.GetEnumerator())
                    {
                        while (enumerator.MoveNext())
                        {
                            var document = enumerator.Current;
                            // evaluating the object
                        }
                    }
                }
            }

Here is the object that I was able to print out:

[21:37:23 INF] CHANGE STREAM DOCUMENT:
**BackingDocument:** ["_id={ \"_data\" : \"825E8A902B000000022B022C0100296E5A1004B805E49E7E3F45C781C4AB942B479670465A5F6964005A100408D7D9CFCB7D62096892F2000172E3CF0004\" }", "operationType=update", "clusterTime=6812415900709289986", **"fullDocument=BsonNull"**, "ns={ \"db\" : \"submission\", \"coll\" : \"SubmissionContainers\" }", "documentKey={ \"_id\" : CSUUID(\"08d7d9cf-cb7d-6209-6892-f2000172e3cf\") }", "updateDescription={ \"updatedFields\" : { \"lastModified\" : \"2020-04-06T02:12:58.7825657\", \"submission\" : { \"effectiveDate\" : \"2020-04-08\", \"workersCompensation\" : { \"employersLiability\" : { \"eachAccident\" : 100000, \"eachEmployee\" : 100000, \"eachPolicy\" : 500000 }, \"legalEntities\" : [{ \"businessType\" : \"LimitedLiabilityCompany\", \"states\" : [{ \"code\" : \"Colorado\", \"experienceModification\" : { \"factor\" : NumberDecimal(\"1\") }, \"locations\" : [{ \"exposure\" : [{ \"payroll\" : 300000, \"class\" : \"9083\", \"rate\" : NumberDecimal(\"0\"), \"hazardGroup\" : \"\\u0000\", \"overrideRate\" : false, \"state\" : \"0\" }], \"fullTimeEmployeeCount\" : 1, \"partTimeEmployeeCount\" : 1, \"address\" : { \"line1\" : \"P.O. Box 100\", \"city\" : \"Broomfield\", \"zip\" : \"80020\", \"state\" : \"Colorado\" } }] }], \"name\" : \"PPTEST: Testing 001\", \"taxId\" : \"121111111\" }] }, \"contacts\" : [{ \"firstName\" : \"ProdIshop\", \"lastName\" : \"Agent\", \"email\" : \"staging+PROD_ishop_agent@pieinsurance.com\", \"type\" : 5 }], \"namedInsured\" : \"PPTEST: Testing 001\" } }, \"removedFields\" : [] }"],
**FullDocument: null,**
DocumentKey: ["_id=UuidStandard:0x08d7d9cfcb7d62096892f2000172e3cf"],
ResumeToken: ["_data=825E8A902B000000022B022C0100296E5A1004B805E49E7E3F45C781C4AB942B479670465A5F6964005A100408D7D9CFCB7D62096892F2000172E3CF0004"],
UpdateDescription: MongoDB.Driver.ChangeStreamUpdateDescription,
CollectionNamespace: submission.SubmissionContainers,
OperationType: Update,
ClusterTime: 6812415900709289986

Most of the time things are working fine, but it seems like there are certain (usually human) operations that happen from developers in the test environments or when manually cleaning up data that causes these null documents on updates - is this just a bug in the C# APIs?

@wan - any ideas or pointers to others who may know?

Hi @Jeremy_Buch,

This is the behaviour from the MongoDB server, not from the C# APIs.

If there are one or more majority-committed operations that modified the updated document after the update operation but before the lookup, the full document returned may differ significantly from the document at the time of the update operation. In this case, it’s null if another operation deletes the document before the lookup operation happens.

The deltas information under updateDescription should still be correct however. See Lookup Full Document for Update Operations for more information.

Regards,
Wan.

2 Likes

Wan,

Thanks for responding. I hadn’t seen this kind of issue other than dev-initiated bulk deletes with robo3t, so it may be changing the object and then deleting it, but either way the scenario is a match for what I’m seeing. I saw the documentation and had assumed that this was not relevant since I had tested this case manually by issuing an insert, a delete, a recreation and an update and the oplog was returning accurate results each step along the way even though I didn’t call it until after all of the operations had been applied. Does this really only show up when the number of deletes is excessive?

At any rate, this is the scenario that I’m seeing since I know it was dev deletes - thanks for confirming since my simple tests early on showed that I could see through deletes for simple operations. Because of this, I wasn’t expecting this to still be an outstanding issue.

Thanks Wan!
Jeremy Buch

Hi @Jeremy_Buch,

It’s not quite about excessiveness, it’s about operations interleaving for the same document. For example:

  • t1 - Update to document A
  • t2 - Document A deleted by another operation
  • t3 - Lookup document A to be returned

Regards,
Wan.

2 Likes

OK - thanks Wan.

I have literally tested exactly that scenario on a small scale and did not see the loss of update information, so I’ll need to repeat that and confirm. I stopped the stream reader, did an update, a delete and a recreate of the same object with different data and then started the stream reader and caught up on records - I was able to see all of the interactions and the accurate state of the object at each step. I saw this note in the documentation before and was concerned by it, so I tested it and didn’t see it exhibit itself.

I’ll re-run my tests to understand it better, but either way it shouldn’t be a blocker for us since we don’t do deletes regularly as part of the workflow (other than manual dev operations).

Thanks Wan!