Keep getting BadChangeset Error

mrackwitz · August 16, 2020, 11:51am

Hey folks,

one of your former Realmies here. Lovely to see btw. that there are still some familiar faces around from back in the day.

Well, I wanted to give MongoDB Realm a spin and use it to build a product myself for once. But I keep hitting walls and I cannot find why.

Context and Stumblings

For context, I’m trying to use the Cocoa bindings from a macOS app in beta-2. I tried the tag which is published on CocoaPods, but saw that there are some not-yet-published improvements on the branch v10, so I hoped maybe if I go bleeding edge it will be better, but no dice. )

I’ve my MongoDB Realm sync in development mode and my sync permissions are true for read and write, which should be as permissive as it gets.

Note:
The UI around could give a little intro in what these expressions actually mean, at least provide a direct links to the docs and the semantics are totally unclear as is. But also the documentation around this is lacking a good intro in what’s going on really. But maybe this is all subject to change?)

I’ve tried to change the type of the partition key, the name of the partition key (_partitionKey and _partition, tried to stay close with the docs ), I read a lot of docs, tutorials, examples, read some code diffs to catch up with what has changed and I could be missing*, … But there is almost nothing around this particular error, which is really frustrating. I diffed my exported server config with the tutorial app, but nothing stand out which should be problematic.

Note:
By reading the cocoa bindings code, I first found out about EmbeddedObject, which is pretty awesome and made me very happy, but it is not even mentioned with a single word in the MongoDB Realm docs. I think it would be good to give an explanation on how this affects sync and maps to MongoDB Atlas. This whole latter part is very opaque right now and it would help a lot, if this would be laid out better to the user imho.

About my error:

I get the following error in the MongoDB Realm UI in the logs:

Ending session with error: failed to validate upload changesets: instruction had incorrect partition value for key "_partition" (ProtocolErrorCode=212)

This maps to what I see in the macOS client in the logs. That is logLevel to .debug, plus one print line custom logging from the error handler. (starting with , as seen in the log attached below) What (I think) I can recognise is that this must be something related to some validation on the server-side, as given by the error class. Partitioning was before my time and I’m sure a lot has changed around that since the MongoDB integration, so I don’t really know what’s going on there. I feel like I’m holding it wrong. Just not sure how, where or why.

Server Log

Summary

BadChangeset Error

Error:

Ending session with error: failed to validate upload changesets: instruction had incorrect partition value for key "_partition" (ProtocolErrorCode=212)

Partition:

master

Session Metrics:

{}

SDK:

Realm Cocoa v10.0.0-beta.2

Platform Version:

Version 10.15.5 (Build 19F101)

Client Log

Summary

2020-08-16 13:22:59.799380+0200 Editor[99805:16209519] Sync: Realm sync client ([realm-core-10.0.0-beta.1], [realm-sync-10.0.0-beta.2])
2020-08-16 13:22:59.799460+0200 Editor[99805:16209519] Sync: Supported protocol versions: 1-1
2020-08-16 13:22:59.799514+0200 Editor[99805:16209519] Sync: Platform: macOS Darwin 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64
2020-08-16 13:22:59.799556+0200 Editor[99805:16209519] Sync: Build mode: Release
2020-08-16 13:22:59.799593+0200 Editor[99805:16209519] Sync: Config param: max_open_files = 256
2020-08-16 13:22:59.799629+0200 Editor[99805:16209519] Sync: Config param: one_connection_per_session = 1
2020-08-16 13:22:59.799667+0200 Editor[99805:16209519] Sync: Config param: connect_timeout = 120000 ms
2020-08-16 13:22:59.799701+0200 Editor[99805:16209519] Sync: Config param: connection_linger_time = 30000 ms
2020-08-16 13:22:59.799733+0200 Editor[99805:16209519] Sync: Config param: ping_keepalive_period = 60000 ms
2020-08-16 13:22:59.799765+0200 Editor[99805:16209519] Sync: Config param: pong_keepalive_timeout = 120000 ms
2020-08-16 13:22:59.799797+0200 Editor[99805:16209519] Sync: Config param: fast_reconnect_limit = 60000 ms
2020-08-16 13:22:59.799829+0200 Editor[99805:16209519] Sync: Config param: disable_upload_compaction = 0
2020-08-16 13:22:59.799860+0200 Editor[99805:16209519] Sync: Config param: tcp_no_delay = 0
2020-08-16 13:22:59.799892+0200 Editor[99805:16209519] Sync: Config param: disable_sync_to_disk = 0
2020-08-16 13:22:59.799927+0200 Editor[99805:16209519] Sync: User agent string: ‘RealmSync/10.0.0-beta.2 (macOS Darwin 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64) RealmObjectiveC/10.0.0-beta.2 $appId’
2020-08-16 13:22:59.801140+0200 Editor[99805:16211279] Sync: Connection[1]: WebSocket::Websocket()
2020-08-16 13:22:59.801243+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Binding ‘$path’ to ‘“master”’
2020-08-16 13:22:59.801299+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Activating
2020-08-16 13:22:59.801370+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: client_reset_config = false, Realm exists = true, async open = false, client reset = false
2020-08-16 13:22:59.801418+0200 Editor[99805:16211279] Sync: Opening Realm file: $path
2020-08-16 13:22:59.801816+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: client_file_ident = 0, client_file_ident_salt = 0
2020-08-16 13:22:59.801892+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Progress handler called, downloaded = 0, downloadable(total) = 0, uploaded = 0, uploadable = 0, reliable_download_progress = 0, snapshot version = 1
2020-08-16 13:22:59.801958+0200 Editor[99805:16211279] Sync: Connection[1]: Resolving ‘ws.eu-west-1.aws.realm.mongodb.com:443’
2020-08-16 13:22:59.831549+0200 Editor[99805:16211279] Sync: Connection[1]: Connecting to endpoint ‘99.83.185.35:443’ (1/2)
2020-08-16 13:22:59.840666+0200 Editor[99805:16211279] Sync: Connection[1]: Connected to endpoint ‘99.83.185.35:443’ (from ‘192.168.178.65:60858’)
2020-08-16 13:22:59.864231+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Progress handler called, downloaded = 0, downloadable(total) = 0, uploaded = 0, uploadable = 2216, reliable_download_progress = 0, snapshot version = 2
2020-08-16 13:22:59.951901+0200 Editor[99805:16211279] Sync: Connection[1]: WebSocket::initiate_client_handshake()
2020-08-16 13:23:00.025554+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Progress handler called, downloaded = 0, downloadable(total) = 0, uploaded = 0, uploadable = 2318, reliable_download_progress = 0, snapshot version = 3
2020-08-16 13:23:00.090092+0200 Editor[99805:16211279] Sync: Connection[1]: WebSocket::handle_http_response_received()
2020-08-16 13:23:00.090171+0200 Editor[99805:16211279] Sync: Connection[1]: Negotiated protocol version: 1
2020-08-16 13:23:00.090222+0200 Editor[99805:16211279] Sync: Connection[1]: Will emit a ping in 20117 milliseconds
2020-08-16 13:23:00.090268+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Sending: BIND(path=‘“master”’, signed_user_token_size=469, need_client_file_ident=1, is_subserver=0)
2020-08-16 13:23:01.349524+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Received: IDENT(client_file_ident=5, client_file_ident_salt=387383682228219102)
2020-08-16 13:23:01.350739+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Sending: IDENT(client_file_ident=5, client_file_ident_salt=387383682228219102, scan_server_version=0, scan_client_version=0, latest_server_version=0, latest_server_version_salt=0)
2020-08-16 13:23:01.350905+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Sending: MARK(request_ident=1)
2020-08-16 13:23:01.605823+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Received: DOWNLOAD(download_server_version=1, download_client_version=0, latest_server_version=1, latest_server_version_salt=3644857074679688014, upload_client_version=0, upload_server_version=0, downloadable_bytes=0, num_changesets=1, …)
2020-08-16 13:23:01.606233+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Finished changeset indexing (incoming: 1 changeset(s) / 6 instructions, local: 2 changeset(s) / 128 instructions, conflict group(s): 2)
2020-08-16 13:23:01.606478+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Finished transforming 2 local changesets through 1 incoming changesets (128 vs 6 instructions, in 2 conflict groups)
2020-08-16 13:23:01.607236+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: 1 remote changeset integrated, producing client version 5
2020-08-16 13:23:01.607313+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Progress handler called, downloaded = 105, downloadable(total) = 105, uploaded = 0, uploadable = 2318, reliable_download_progress = 1, snapshot version = 5
2020-08-16 13:23:01.748902+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Received: MARK(request_ident=1)
2020-08-16 13:23:01.749017+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Sending: UPLOAD(progress_client_version=5, progress_server_version=1, locked_server_version=1, num_changesets=2)
2020-08-16 13:23:01.749137+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Upload compaction: original size = 2216, compacted size = 2216
2020-08-16 13:23:01.749204+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Upload compaction: original size = 102, compacted size = 102
2020-08-16 13:23:01.790901+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Received: ERROR(error_code=212, message_size=22, try_again=0)
2020-08-16 13:23:01.791019+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Suspended
2020-08-16 13:23:01.791269+0200 Editor[99805:16211279] Sync: Connection[1]: Session[1]: Sending: UNBIND
Unhandled sync error: io.realm.sync:4 Bad changeset (UPLOAD)2020-08-16 13:23:01.791541+0200 Editor[99805:16211279] Sync: Connection[1]: Disconnected

Jay · August 16, 2020, 1:34pm

TL;DR

I’ve posted information below however, we have an existing project that has remained unchanged code or data wise for two weeks and are getting a very similar error BadChangeset Error however, the trailing info is different - your error is related to the partition key (read below), ours is related to ERROR: AddColumn when we didn’t make any changes to our objects, data or code.

This may be server related and I have opened a ticket and will post with any updates.

It would be helpful to include the specific code that causes the error along with how your objects are defined and then what change was made to those objects. e.g. my objects had a partition key of @objc dynamic var _partitionKey and I changed that to @objc dynamic var _super_partitionKey

Also this

I’ve tried to change the type of the partition key,

Is not possible in that context without jumping through a bunch of hoops. General rule of thumb is; once it’s set, it’s set. Attempting to change it without the other steps would explain the error

instruction had incorrect partition value for key “_partition”

A few notes

Realm clients should never modify the partition value directly. Any field that clients can modify cannot be used as a partition key.
Changing a partition value is an expensive operation, requiring a deletion of the object from one realm and an insertion of the same object into another realm. As a result, you should choose a partition key that rarely, if ever, needs to change.
Once you have chosen a partition key and enabled sync, you cannot reassign the partition key to a different field. To use a different field as your partition key you’ll need to stop sync, change it, re-enable sync and then deal with a client reset in code (this goes back to bullet point #2 in that the server does a lot of work on the back end when this is changed)

A good overview is in the Partition Documentation

Ian_Ward · August 16, 2020, 7:57pm

Bad Changeset is an error in the low-level C++ sync-client and cannot be handled by the upper level binding code. I have upped the priority of this issue and we are working on a fix this week.

mrackwitz · August 16, 2020, 8:21pm

Hey @Ian_Ward, long time! Awesome that this is going to be fixed soon.
Any workarounds for now? This is pretty much a complete showstopper for me atm.

mrackwitz · August 16, 2020, 8:30pm

Thanks @Jay for your quick reply.

Is not possible in that context without jumping through a bunch of hoops. General rule of thumb is; once it’s set, it’s set. Attempting to change it without the other steps would explain the error

To use a different field as your partition key you’ll need to stop sync, change it, re-enable sync and then deal with a client reset in code (this goes back to bullet point #2 in that the server does a lot of work on the back end when this is changed)

I know, I’ve jumped at least thru the hoops you describe later (as in 2nd quote) on now several times. Very familiar with the procedure to nuke the local Realm as a solution…
If there are any more steps involved please let me know.

My partition key is defined as follows:

@objc dynamic var _partition: String?

Also tried:

@objc dynamic var _partition: String = ""

And:

@objc dynamic var _partitionKey: String = ""

With sync terminations, rule changes, sync re-start, nuking local Realm and re-building and starting the app in between, obviously.

Ian_Ward · August 16, 2020, 8:36pm

@mrackwitz For now, you can probably avoid this issue by disabling Developer mode on the server side. You would then need to define your syncing schema in JSON on the cloud though. You would also need to wipe your client to start fresh.

mrackwitz · August 16, 2020, 8:54pm

Thanks for the super fast response. I tried this one too. I disabled dev mode, reduced my schema to a very limited subset, but the error still persists.

Ian_Ward · August 16, 2020, 9:02pm

@mrackwitz Did you wipe your simulator as well?

mrackwitz · August 17, 2020, 7:57am

It’s a macOS app. I did a “rm -rf” on “…/Application Support/realm-object-server/”. That should wipe it all?

nirinchev · August 17, 2020, 8:06am

Hey Marius, instruction had incorrect partition value for key "_partition" occurs when the value for the partition key is different from the value of the partition with which you opened the Realm.

So something like:

realm = try! Realm(configuration: user.configuration(partitionValue: "abc"))
try! realm.write {
  realm.add(Foo(_partition: "def"))
}

Note that the Realm was opened with a partition value of abc, but the _partition field is assigned to def. To avoid the error you can either assign the _partition field in your model to the correct value or avoid assigning it to anything (that way the server will populate it). If you don’t care about the _partition field in your mobile app, then you can just remove it from your swift model altogether - technically it will still be in the database, because the server will synchronize it to the device, but not having it defined in your models will ensure you don’t accidentally assign it to the wrong value.

mrackwitz · August 17, 2020, 8:23am

Heyy @nirinchev, thanks for chiming in!

To avoid the error you can either assign the _partition field in your model to the correct value

I tried this, that did not seem to work for me with a string partition key. The partition key I attempted to use was just "master".

or avoid assigning it to anything (that way the server will populate it).

That’s what I tried next. (Possibly default values as they are used in the Cocoa / Swift SDK might got in between here. For the record, I used the constructor of the object with zero arguments directly and set all fields externally, beside the partition key. I tried this with a required string and an optional string as partition key.)

If you don’t care about the _partition field in your mobile app, then you can just remove it from your swift model altogether - technically it will still be in the database, because the server will synchronize it to the device, but not having it defined in your models will ensure you don’t accidentally assign it to the wrong value.

Well, that worked finally. Knowing about this earlier could have saved me some pain and time, would be nice to include this in the docs.

In fact, I just looked it up again, the docs are then even completely misleading here, they say Realm Sync would ignore Collections without partition key:

However, you don’t have to include the partition key in all collections; Realm Sync ignores any collections that lack a Realm Schema as well as any documents that lack a valid value for the partition key.

My expectation from reading this was, if I don’t include a partition key on any particular object, those objects won’t be synced at all.

nirinchev · August 17, 2020, 8:39am

Hm… I wonder if there’s an issue with the Cocoa SDK where it doesn’t encode the string value correctly. I’ll file a ticket for the server team to log the expected and the actual values for the partition key to make it easier to spot mistakes. Once that is there, I’ll see what’s going on with the Cocoa SDK and why there’s a discrepancy with the partition keys. Note that if you had default values, it’s very likely that this was the cause - the way Sync will encode this is: one instruction to set _partition to its default value, then a second instruction to set _partition to "master". In this case, the first instruction would violate the validations and the entire changeset will be rejected, even though the final value of the field is correct. I’ll check with the Cocoa team to confirm if we can do anything about that.

I can see how that would be confusing - this explanation actually targets the server-side schema and concerns synchronization from MongoDB to Realm - i.e. if you have collections in MongoDB that you don’t want to sync to the mobile device, you can omit the partition key from their schema and sync will ignore them. At the moment all objects from the mobile database get synchronized to the server, but we may consider some opt-out mechanism in the future.

In any case, great to hear that things are working for you and thanks for the valuable feedback. I’ll make sure to convey it to the right people so that future users get a more seamless experience

mrackwitz · August 17, 2020, 4:11pm

That would make sense and was along the lines, I was thinking. If Cocoa, can’t do anything about it, maybe making up in the documentation or even in the error message itself, could probably help a lot.

Working, but not for long actually. Just after a subsequent start, I get failures again, same error, but different reasoning. Quite reproducible again tho unfortunately.

failed to validate upload changesets: clients cannot upload destructive schema changes, received "EraseColumn" instruction

Any tips what the culprit could be in that case?
Note: No, I’m not actually doing any schema changes at all.
In fact I’ve tried to terminate sync and start again, and it does sync even in development mode, where this error is happening as well.
Could this have something to do with that the server rules include the partition key, but the client schema doesn’t?

nirinchev · August 17, 2020, 5:42pm

That’s super interesting - the EraseColumn instruction should be impossible to generate from the SDK. I’m sorry you’re hitting these issues, but the feedback is very valuable. I’ll try to isolate a repro case, but just to be sure, my understanding is that these are the steps you’re taking:

Define some schema on the server.
Define matching models in your client app.
Remove the _partition field from your client schema.
Connect to the server, sync some data.
Restart your app.
Get the destructive schema change error.

Does this sound reasonable?

mrackwitz · August 17, 2020, 5:56pm

Well, glad that the feedback at least is helpful. I can keep it coming, as I keep hitting issues, once this one is resolved.

To the repro steps: this does indeed sound like what I’ve done, but little disclaimer: once sync was initially working I opened the flood gates and added my fully fledged model, instead of the previously very simplified version I’ve used. That alone was an additive change though, but there is a chance that there is more going on. Nevertheless given that I didn’t make any subsequent changes to the schema, the partition key should be indeed the only difference which could be somehow interpreted as destructive.

Keep me posted on ideas what I can do to resolve this issue, or at least side-step this issue again. Meanwhile I’m starting to seriously consider to just use at least the Mongo DB document-driven API with plain JSON for now, but that would require implementing JSON mapping and all its drawbacks afaict.

nirinchev · August 17, 2020, 8:12pm

Haven’t had much luck reproing it, but have you tried wiping your local state? While the bug is real and we’ll be hunting it, I imagine that should unblock you in the short term.

mrackwitz · August 18, 2020, 7:49am

Several times.

I was still on the v10 branch, so theoretically there could be something in there? But I just checked and saw that @jsflx published the latest changes just yesterday as v10.0.0-beta.3, so I might as well use that.

Ian_Ward · August 19, 2020, 3:50pm

@mrackwitz Would you be able to send me a repro case so that I can recreate it internally? You can email me at ian.ward@mongodb.com

Barry_Fawthrop · August 21, 2020, 6:42pm

I’m getting a very similar error but coming from Android/Kotlin
failed to validate upload changesets: instruction had incorrect partition value for key “username” (ProtocolErrorCode=212)

Yet username is assigned in the class Object and it has a valid value

Ian_Ward · August 21, 2020, 6:44pm

@Barry_Fawthrop Please share your server side logs - what do they say for this error?