Sync performance

After running a script to load data into a synced MondoDB Realm and the script eventually failing with a “Bad sync process (7)” error after the data load has completed but before the sync has completed I restart the script in query mode and the sync seems to attempt to pick up where it left off but seems to be doing some additional steps and is very very slow - it has been running for a day now.

Can anyone explain what the sync is doing when “Transforming local changeset(s)…” and why it might be so slow ?

SYNC: [2] Realm sync client ([realm-core-10.1.4], [realm-sync-10.1.5])
SYNC: [2] Supported protocol versions: 2-2
SYNC: [2] Platform: macOS Darwin 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec  2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 x86_64
SYNC: [2] Build mode: Release
SYNC: [2] Config param: max_open_files = 256
SYNC: [2] Config param: one_connection_per_session = 1
SYNC: [2] Config param: connect_timeout = 120000 ms
SYNC: [2] Config param: connection_linger_time = 30000 ms
SYNC: [2] Config param: ping_keepalive_period = 60000 ms
SYNC: [2] Config param: pong_keepalive_timeout = 120000 ms
SYNC: [2] Config param: fast_reconnect_limit = 60000 ms
SYNC: [2] Config param: disable_upload_compaction = 0
SYNC: [2] Config param: tcp_no_delay = 0
SYNC: [2] Config param: disable_sync_to_disk = 0
SYNC: [2] User agent string: 'RealmSync/10.1.5 (macOS Darwin 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec  2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 x86_64) RealmJS/10.1.2 (node.js, darwin, vv12.20.0) '
SYNC: [2] Connection[1]: WebSocket::Websocket()
SYNC: [3] Connection[1]: Session[1]: Binding '/Users/duncangroenewald/Development/RealmMigrationMongoDB/mongodb-realm/makespace-development-qkudm/5fffc37714c08f5936e29aef/s_default.realm' to '"default"'
SYNC: [2] Connection[1]: Session[1]: Activating
SYNC: [4] Connection[1]: Session[1]: client_reset_config = false, Realm exists = true, async open = false, client reset = false
SYNC: [2] Opening Realm file: /Users/duncangroenewald/Development/RealmMigrationMongoDB/mongodb-realm/makespace-development-qkudm/5fffc37714c08f5936e29aef/s_default.realm
SYNC: [2] Connection[1]: Session[1]: client_file_ident = 1, client_file_ident_salt = 301361383067496991
SYNC: [1] Connection[1]: Session[1]: last_version_available  = 350
SYNC: [1] Connection[1]: Session[1]: progress_server_version = 68
SYNC: [1] Connection[1]: Session[1]: progress_client_version = 50
SYNC: [1] Using already open Realm file: /Users/duncangroenewald/Development/RealmMigrationMongoDB/mongodb-realm/makespace-development-qkudm/5fffc37714c08f5936e29aef/s_default.realm
SYNC: [2] Connection[1]: Session[1]: Progress handler called, downloaded = 3670, downloadable(total) = 3670, uploaded = 15004029, uploadable = 79150774, reliable_download_progress = 0, snapshot version = 350
SYNC: [3] Connection[1]: Resolving 'ws.realm.mongodb.com:443'
SYNC: [3] Connection[1]: Connecting to endpoint '52.64.157.195:443' (1/1)
SYNC: [4] Connection[1]: Connected to endpoint '52.64.157.195:443' (from '10.0.1.171:50581')
SYNC: [2] Connection[1]: WebSocket::initiate_client_handshake()
SYNC: [1] Connection[1]: HTTP request =
GET /api/client/v2.0/app/makespace-development-qkudm/realm-sync HTTP/1.1
Host: ws.realm.mongodb.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJiYWFzX2RldmljZV9pZCI6IjVmZmZmMDJmNTczNGRmNWQyNzQ2NGY4OSIsImJhYXNfZG9tYWluX2lkIjoiNWZmZmMyZjA2YzEwMjRmZmU3ZDIzYjYwIiwiZXhwIjoxNjEwNjEwNDg4LCJpYXQiOjE2MTA2MDg2ODcsImlzcyI6IjVmZmZmMDJmNTczNGRmNWQyNzQ2NGY5YiIsInN0aXRjaF9kZXZJZCI6IjVmZmZmMDJmNTczNGRmNWQyNzQ2NGY4OSIsInN0aXRjaF9kb21haW5JZCI6IjVmZmZjMmYwNmMxMDI0ZmZlN2QyM2I2MCIsInN1YiI6IjVmZmZjMzc3MTRjMDhmNTkzNmUyOWFlZiIsInR5cCI6ImFjY2VzcyJ9.VzDFU9zouyEqEYAPCRkZOga4KYnfjOwhKhI4_hQF6B0
Connection: Upgrade
Sec-WebSocket-Key: hSprx0sKupZR/g39vOZkmw==
Sec-WebSocket-Protocol: com.mongodb.realm-sync/2
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: RealmSync/10.1.5 (macOS Darwin 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec  2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 x86_64) RealmJS/10.1.2 (node.js, darwin, vv12.20.0)


SYNC: [2] Connection[1]: WebSocket::handle_http_response_received()
SYNC: [1] Connection[1]: HTTP response = HTTP/1.1 101 Switching Protocols
cache-control: no-cache, no-store, must-revalidate
connection: Upgrade
date: Thu, 14 Jan 2021 07:18:08 GMT
sec-websocket-accept: I7CDbiLyq7SKRXjK9JaK8hIaMxk=
sec-websocket-protocol: com.mongodb.realm-sync/2
server: envoy
upgrade: websocket
vary: Origin
x-frame-options: DENY


SYNC: [3] Connection[1]: Negotiated protocol version: 2
SYNC: [2] Connection[1]: Will emit a ping in 48083 milliseconds
SYNC: [2] Connection[1]: Session[1]: Sending: BIND(path='"default"', signed_user_token_size=469, need_client_file_ident=0, is_subserver=0)
SYNC: [2] Connection[1]: Session[1]: Sending: IDENT(client_file_ident=1, client_file_ident_salt=301361383067496991, scan_server_version=68, scan_client_version=50, latest_server_version=68, latest_server_version_salt=7290643098720209980)
SYNC: [2] Connection[1]: Session[1]: Sending: MARK(request_ident=2)
SYNC: [1] Connection[1]: Download message compression: is_body_compressed = 1, compressed_body_size=484573, uncompressed_body_size=1491642
SYNC: [1] Connection[1]: Received: DOWNLOAD CHANGESET(server_version=75, client_version=52, origin_timestamp=190527672689, origin_file_ident=3, original_changeset_size=610, changeset_size=610)
SYNC: [1] Connection[1]: Changeset: 3F 00 07 41 70 70 55 73 65 72 3F 01 24 33 35 35 35 42 44 44 34 2D 43 39 45 38 2D 34 35 38 34 2D 39 30 30 42 2D 46 46 35 39 37 35 46 37 35 38 35 33 3F 02 0D 5F  33 36 36 34 30 30 32 31 43 35 3F...
SYNC: [1] Connection[1]: Received: DOWNLOAD CHANGESET(server_version=77, client_version=52, origin_timestamp=190527672928, origin_file_ident=3, original_changeset_size=266399, changeset_size=266399)
SYNC: [1] Connection[1]: Changeset: 3F 00 0A 41 73 73 6F 72 74 6D 65 6E 74 3F 01 24 33 39 46 39 39 37 35 32 2D 33 41 45 41 2D 34 31 44 32 2D 41 44 30 43 2D 37 42 33 33 34 32 36 33 41 43 30 44 3F  2D 39 42 38 44 2D 34 43 30 35 2D 39 42 43 34 2D 42 41 46 41 37 36 34 34 39 46 34 42 3F 23 24 32 44 42 44 43 38 33 30 2D 43 36 38...
SYNC: [1] Connection[1]: Received: DOWNLOAD CHANGESET(server_version=78, client_version=52, origin_timestamp=190527674683, origin_file_ident=3, original_changeset_size=1217095, changeset_size=1217095)
SYNC: [1] Connection[1]: Changeset: 3F 00 0E 41 73 73 6F 72 74 6D 65 6E 74 49 74 65 6D 3F 01 24 45 44 37 45 33 30 38 42 2D 41 42 36 46 2D 34 38 36 41 2D 38 42 33 43 2D 45 46 37 33 33 38 38 44 39 6 38 2D 39 36 38 38 2D 45 30 43 36 38 30 42 38 44 39 45 46 3F 27 24 41 39 42 43 30 37 31 38 2D 38 35 34 41 2D 34 31 36 41 2D 42...
SYNC: [2] Connection[1]: Session[1]: Received: DOWNLOAD(download_server_version=78, download_client_version=52, latest_server_version=78, latest_server_version_salt=2596012816928273966, upload_client_version=52, upload_server_version=0, downloadable_bytes=0, num_changesets=4, ...)
SYNC: [1] Using already open Realm file: /Users/duncangroenewald/Development/RealmMigrationMongoDB/mongodb-realm/makespace-development-qkudm/5fffc37714c08f5936e29aef/s_default.realm
SYNC: [1] Connection[1]: Session[1]: Scanning incoming changeset [1/4] (32 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning incoming changeset [2/4] (486 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning incoming changeset [3/4] (18970 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning incoming changeset [4/4] (84042 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [1/192] (5652 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [2/192] (20010 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [3/192] (20010 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [4/192] (20010 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [5/192] (20010 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [6/192] (20010 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [7/192] (12920 instructions)
...
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [190/192] (3044 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [191/192] (58 instructions)
SYNC: [1] Connection[1]: Session[1]: Scanning local changeset [192/192] (217 instructions)
SYNC: [1] Connection[1]: Session[1]: Indexing incoming changeset [1/4] (32 instructions)
SYNC: [1] Connection[1]: Session[1]: Indexing incoming changeset [2/4] (486 instructions)
SYNC: [1] Connection[1]: Session[1]: Indexing incoming changeset [3/4] (18970 instructions)
SYNC: [1] Connection[1]: Session[1]: Indexing incoming changeset [4/4] (84042 instructions)
SYNC: [2] Connection[1]: Session[1]: Finished changeset indexing (incoming: 4 changeset(s) / 103530 instructions, local: 192 changeset(s) / 2404763 instructions, conflict group(s): 66)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [1/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [2/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [3/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [4/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [5/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [6/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [7/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [8/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [9/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [10/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [11/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [12/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [13/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [14/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [15/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [16/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [17/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [18/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [19/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [20/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [21/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [22/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [23/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [24/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [25/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [26/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [27/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [28/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [29/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [30/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [31/192] through 4 incoming changeset(s) with 66 conflict group(s)
SYNC: [1] Connection[1]: Session[1]: Transforming local changeset [32/192] through 4 incoming changeset(s) with 66 conflict group(s)

Previously when running the load script it would always fail with the “Bad sync process (7)” error but when restarted in query mode the sync would usually continue uploading change sets until it received another “Bad sync process (7)” error. Enough restarts and the sync would eventually complete.

Now the behaviour seems to have changed.

Any ideas as to what is going on here ?