Unable to upgrade from 3.4.23 to 3.6.17 -

Hi,

I’m having issues upgrading a Debian 9.11 (Stretch) server with MongoDB 3.4.23 to 3.6.17. MongoDB 3.4.23 is from the Jessie repository and 3.6.17 is in the Stretch repository

The cluster is a ‘simple’ 3 node cluster, all with mongod running happily and having been upgraded continuously from the 2.2 days

The upgrade of a secondary appears to go well, until mongod starts, at which point it explodes with an error about DBException::toString(): Location40415: BSON field ‘MinValidDocument.h’ is an unknown field. This happens on any secondary I try to upgrade, and is also reverted reasonably well by just downgrading back to 2.4.23

I’ve replicated this in a Vagrant test cluster, with a copy of our database, but am at a loss as to how to proceed to fix this. Any input would be appreciated.

Cheers

Kyle

2020-02-05T12:04:12.148+0000 I CONTROL [signalProcessingThread] now exiting
2020-02-05T12:04:12.148+0000 I CONTROL [signalProcessingThread] shutting down with code:0
2020-02-05T12:04:58.379+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] MongoDB starting : pid=19342 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb-2
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] db version v3.6.17
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] modules: none
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] build environment:
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] distmod: debian92
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] distarch: x86_64
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] options: { config: “/etc/mongod.conf”, net: { bindIp: “192.168.56.102”, port: 27017 }, replication: { replSetName: “linndb” }, storage: { dbPath: “/var/lib/mongodb”, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, path: “/var/log/mongodb/mongod.log” } }
2020-02-05T12:04:58.394+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the ‘mmapv1’ storage engine, so setting the active storage engine to ‘mmapv1’.
2020-02-05T12:04:58.403+0000 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal
2020-02-05T12:04:58.403+0000 I JOURNAL [initandlisten] recover : no journal files present, no recovery needed
2020-02-05T12:04:58.404+0000 I JOURNAL [durability] Durability thread started
2020-02-05T12:04:58.405+0000 I JOURNAL [journal writer] Journal writer thread started
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten]
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten]
2020-02-05T12:04:58.913+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory ‘/var/lib/mongodb/diagnostic.data’
2020-02-05T12:04:58.984+0000 I REPL [initandlisten] Did not find local initialized voted for document at startup.
2020-02-05T12:04:58.985+0000 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one.
2020-02-05T12:04:58.985+0000 I STORAGE [initandlisten] createCollection: local.system.rollback.id with no UUID.
2020-02-05T12:04:58.986+0000 I REPL [initandlisten] Initialized the rollback ID to 1
2020-02-05T12:04:59.032+0000 F REPL [initandlisten] Caught exception during replication recovery: Location40415: BSON field ‘MinValidDocument.h’ is an unknown field.
2020-02-05T12:04:59.032+0000 F - [initandlisten] terminate() called. An exception is active; attempting to gather more information
2020-02-05T12:04:59.055+0000 F - [initandlisten] DBException::toString(): Location40415: BSON field ‘MinValidDocument.h’ is an unknown field.
Actual exception type: mongo::error_details::throwExceptionForStatus(mongo::Status const&)::NonspecificAssertionException

0x55bb915f6511 0x55bb915f5ef5 0x55bb916ea486 0x55bb916ea4d1 0x55bb90183cba 0x55bb904afadf 0x55bb904b027f 0x55bb8fd81d94 0x55bb8fd83db2 0x55bb8fd07ec9 0x7fae51a382e1 0x55bb8fd6c0da
----- BEGIN BACKTRACE -----
{“backtrace”:[{“b”:“55BB8F36B000”,“o”:“228B511”,“s”:"_ZN5mongo15printStackTraceERSo"},{“b”:“55BB8F36B000”,“o”:“228AEF5”},{“b”:“55BB8F36B000”,“o”:“237F486”,“s”:"_ZN10__cxxabiv111__terminateEPFvvE"},{“b”:“55BB8F36B000”,“o”:“237F4D1”},{“b”:“55BB8F36B000”,“o”:“E18CBA”,“s”:"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextE"},{“b”:“55BB8F36B000”,“o”:“1144ADF”,“s”:"_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE"},{“b”:“55BB8F36B000”,“o”:“114527F”,“s”:"_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE"},{“b”:“55BB8F36B000”,“o”:“A16D94”},{“b”:“55BB8F36B000”,“o”:“A18DB2”,“s”:“ZN5mongo11mongoDbMainEiPPcS1”},{“b”:“55BB8F36B000”,“o”:“99CEC9”,“s”:“main”},{“b”:“7FAE51A18000”,“o”:“202E1”,“s”:"__libc_start_main"},{“b”:“55BB8F36B000”,“o”:“A010DA”,“s”:"_start"}],“processInfo”:{ “mongodbVersion” : “3.6.17”, “gitVersion” : “3d6953c361213c5bfab23e51ab274ce592edafe6”, “compiledModules” : , “uname” : { “sysname” : “Linux”, “release” : “4.9.0-11-amd64”, “version” : “#1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)”, “machine” : “x86_64” }, “somap” : [ { “b” : “55BB8F36B000”, “elfType” : 3, “buildId” : “E93AC4BD48D98B49945C22FA72B32037154B506B” }, { “b” : “7FFCC75E2000”, “path” : “linux-vdso.so.1”, “elfType” : 3, “buildId” : “E4710F184535B7465A293C8D84196F236A12D2CF” }, { “b” : “7FAE53001000”, “path” : “/lib/x86_64-linux-gnu/libresolv.so.2”, “elfType” : 3, “buildId” : “EAD5FD817712E63C1212D1EE7D7EE1B9C29F93A7” }, { “b” : “7FAE52B67000”, “path” : “/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1”, “elfType” : 3, “buildId” : “BC8265C9A7EB28F1A9677B7437E4561F383C44D9” }, { “b” : “7FAE528FB000”, “path” : “/usr/lib/x86_64-linux-gnu/libssl.so.1.1”, “elfType” : 3, “buildId” : “4EAE4DD6DA20F6CECBEA6688C8ED8E9987CD800F” }, { “b” : “7FAE526F7000”, “path” : “/lib/x86_64-linux-gnu/libdl.so.2”, “elfType” : 3, “buildId” : “DB2CAEEEC37482A98AB1416D0A9AFE2944930DE9” }, { “b” : “7FAE524EF000”, “path” : “/lib/x86_64-linux-gnu/librt.so.1”, “elfType” : 3, “buildId” : “86B35D63FACD97D22973E99EE9863F7714C4F53A” }, { “b” : “7FAE521EB000”, “path” : “/lib/x86_64-linux-gnu/libm.so.6”, “elfType” : 3, “buildId” : “4E49714C557CE0472C798F39365CA10F9C0E1933” }, { “b” : “7FAE51FD4000”, “path” : “/lib/x86_64-linux-gnu/libgcc_s.so.1”, “elfType” : 3, “buildId” : “51AD5FD294CD6C813BED40717347A53434B80B7A” }, { “b” : “7FAE51DB7000”, “path” : “/lib/x86_64-linux-gnu/libpthread.so.0”, “elfType” : 3, “buildId” : “16D609487BCC4ACBAC29A4EAA2DDA0D2F56211EC” }, { “b” : “7FAE51A18000”, “path” : “/lib/x86_64-linux-gnu/libc.so.6”, “elfType” : 3, “buildId” : “775143E680FF0CD4CD51CCE1CE8CA216E635A1D6” }, { “b” : “7FAE53218000”, “path” : “/lib64/ld-linux-x86-64.so.2”, “elfType” : 3, “buildId” : “606DF9C355103E82140D513BC7A25A635591C153” } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55bb915f6511]
mongod(+0x228AEF5) [0x55bb915f5ef5]
mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x55bb916ea486]
mongod(+0x237F4D1) [0x55bb916ea4d1]
mongod(_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextE+0xBDA) [0x55bb90183cba]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x52F) [0x55bb904afadf]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE+0x1DF) [0x55bb904b027f]
mongod(+0xA16D94) [0x55bb8fd81d94]
mongod(ZN5mongo11mongoDbMainEiPPcS1+0x872) [0x55bb8fd83db2]
mongod(main+0x9) [0x55bb8fd07ec9]
libc.so.6(__libc_start_main+0xF1) [0x7fae51a382e1]
mongod(_start+0x2A) [0x55bb8fd6c0da]
----- END BACKTRACE -----

Hi @Kyle_Gordon,

The issue is due to stricter checking of the fields in the replset.minvalid collection in 3.6+. While the node is running with a 3.4 mongod binary, you can address this issue by running the following command while connected directly to your secondaries:

db.getSiblingDB("local").replset.minvalid.update({},{$unset:{h:"",op:"",ns:"",o:""}})

This should allow the node to startup successfully with a 3.6+ binary.

4 Likes

Similar, maybe identical, problem at https://stackoverflow.com/questions/59098649/mongodb-sharded-cluster-upgrade-from-3-4-23-to-3-6-12-problem

@Kyle_Gordon I would recommend doing what Alex suggested - let us know if that doesn’t resolve your upgrade issue.

@alexbevi @Asya_Kamsky That’s super, thank you. I’ve applied it to the Vagrant test cluster and it’s upgraded successfully. I’ll update the development and production clusters next week.

In the meantime, is this a setting that can safely be left on the clusters?

Yes, you can safely leave the replset.minversion in this state. The issue you experienced is due to stricter checking, so if it passes validation in 3.6 then you will not be able to get your cluster into this state again moving forward.

1 Like