Understanding Partition Keys

I am a little confused about partition keys to get MongoDB Realm and Atlas working together. In particular, I am referencing the material at https://docs.mongodb.com/realm/sync/partitioning/

Whilst there are a couple of examples in the documentation, my use case is a little different. My users create documents - notes, and often they will share those notes (with either read or write permissions) to one or more other users. They may also not share with any other users, likewise, they may also add new shares with additional users or un-share existing users during the lifecycle of a notes document.

The Team Realms example wouldn’t suit here because the users that a document is shared with is not always going to be consistent across documents. So having this alternative teams collection would not suit.

So my thought process…

I currently have two fields in a notes document - sharedViewers and sharedEditors that contain the relevant user IDs where the document is shared with other users.

But I’m then confused about how to deal with the partition key and the realms that are then created locally on users devices.

In addition I noted that the documentation states that:

Avoid changing a partition value in a Realm object using the Realm SDK because it triggers a client reset.

  1. So could this client reset be an issue?
  2. If this approach then creates n number of realms every time documents are shared with a user, then querying across all realms is not possible correct?

Any help would be appreciated as I’m surely not the first that is trying to deal with this type of use case.

1 Like

Hi Anthony,

I found this useful in this matter.

Thanks @Benjamin_Storrier
I did see that and have posted a follow up comment there also because it’s still not 100% clear to me. I really want to use MongoDB as my backend but this ‘complexity’ is leaving me doubtful.

Don’t worry. It’ll grok.
The docs are pretty good. Keep re-reading them and trying things out.
I found the node example to be helpful - try installing and running that.
Take note that the walk through is not complete and you’ll need to refer to the readme on the github.

Thanks. I’ve gone through the tutorials etc. And have watched countless hours of MongoDB videos from past keynotes etc. Some of the documentation is a little unclear between partitioning and then permissions.

@Anthony_CJ I guess the way to think about it is that Realm has now become a mapping on a mapping. The first mapping function is Altas itself that maps objects into collections that conform to a schema. This by the way is similar to the way a Realm Cloud instance worked in the old Realm Cloud product. With MongoDB Realm, the partition key value introduces a second mapping. So basically all objects in all collections with a specific partition key value map into a specific Realm. At first it was a little confusing, but now I have the swing of it.

2 Likes

Thanks @Richard_Krueger. That makes sense actually. So if I’m understanding this correctly, can’t I just map MongoDB Realms by setting the partition key to documents in Atlas where a userID is listed in the documents sharedViewers? That way, as the users are removed from or added to sharedViewers, the realms are ‘updated’?

@Anthony_CJ so basically MongoDB Atlas organizes a data base as set of collections that contain objects that conform to a specific schema. This is the way Realm used to do it before the merger with MongoDB. But with the new system, you have Realm on the front end and MongoDB Atlas on the back end. Since the two models are not completely isomorphic, MongoDB needed a way to map Realm onto Atlas. The way this is done in through a partition key. For a specific app, the developer defines a partition key; usually this is called _partition or _realmId, but it can be called anything you want. There is only one partition per Realm app. The partition key is a property that is defined in the collections in Atlas that Realm maps on to. It’s the partition key value that specifically defines which Realm the object in the collection belongs to. For example, your app might want a Realm called ‘shared_object_realm’, which would be readable by all users. In that case the partition key value would be ‘shared_object_realm’. Similary, the app might want a realm that is only readable/writable by the logged in user and no one else, that partition key value would be the user id of the logged in user. In the older Realm world, this was called a private user realm. The read/write privileges for Realms are controlled through the sync partitions.

I hope this was useful

Richard Krueger

2 Likes

Thanks @Richard_Krueger. It does. What I’m struggling to wrap my head around is a scenario where…
User_A, User_B, User_C
Each can see the documents they’ve created:
Document_1 - created by User_A
Document_2 - created by User_B
Document_3 - created by User_C

So if I had the partition key set to who created the documents, then the realm on each users device has their items. Got that.

But if User_A wants to share Document_1 with User_B. I’m struggling with that concept. I had throughout I could just have a field ‘sharedWith’ and in that field I would store the creating user_id as well as anyone they’ve shared with. So for Document_1 it would hold User_A and User_B that that ‘sharedWith’ field would be the realm sync partition key but I can’t do that. So struggling with the concept of how to structure things in a way that allows documents to be randomly (as selected by a user) shared with 0…n other users.

1 Like

@Anthony_CJ I apologize for having taken an hiatus from these forums lately. My regular 9-5 job has had me buried in a documentation effort, away from programming, for the last two weeks. Let me give a stab at your problem.

At present, MongoDB Realm still has not implemented fine grain rules permissions, so there is no way to give read/write permissions to a particular document to a particular set of users. I am sure that they will get this feature in, but for the moment you have to rely on SYNC level permissions to achieve the same goal.

What you can do now is control whether a user has access to a specific partition key through the use of custom data associated with the user, and SYNC based rules that access that custom data. The custom data would contain a list of all the partitions a user can read from and/or write to (perhaps two separate lists). So coming back to your problem.

Let’s say you have Document_1 that you want to share between User_A and User_B, you would create another partition key value named Team_AB and assign it to the document. You would then include Team_AB in the custom data list for both User_A and User_B. Maybe User_A would have write permission and User_B would only have read permission. I know this is clunky, but it is a work around that works right now.

1 Like

Thanks @Richard_Krueger. I understand we aren’t all on here all the time. But I appreciate your help.

So putting aside the permissions aspect, the actual realm partition syncing is still problematic for me. (sorry)

So with these collections:

collection user: [
   { user_id : "user_1", displayName : "John", email : "john@test.com" }
   { user_id : "user_2", displayName : "Jane", email : "jane@test.com" }
   { user_id : "user_3", displayName : "Adam", email : "adam@test.com" }   
]

collection document: [
   { createdBy: user_1, title : "Document A", sharedWith: [user_1] }
   { createdBy: user_1, title : "Document B", sharedWith: [user_1, user_2] }
   { createdBy: user_2, title : "Document C", sharedWith: [user_2, user_1] }
]

The desired realm results would be as follows:

realm user_1: [
   { user_id : "user_1", displayName : "John", email : "john@test.com" }
   { createdBy: user_1, title : "Document A", sharedWith: [user_1] }
   { createdBy: user_1, title : "Document B", sharedWith: [user_1, user_2] }
   { createdBy: user_2, title : "Document C", sharedWith: [user_2, user_1] }
]

realm user_2: [
   { user_id : "user_2", displayName : "Jane", email : "jane@test.com" }
   { createdBy: user_1, title : "Document B", sharedWith: [user_1, user_2] }
   { createdBy: user_2, title : "Document C", sharedWith: [user_2, user_1] }
]

realm user_3: [
   { user_id : "user_3", displayName : "Adam", email : "adam@test.com" }   
]

Is there a way to do this so that the partition key could be the sharedWith? That way, as users add or remove users from a documents sharing privileges, then the realm syncing would update accordingly.

I get what you are attempting to do here. This is not an answer but did you read through the Define Sync Permissions guide? The function rules section may let you provide/define user access based on a function that evaluates if that use has permission or not.

We poked around with it a bit last week and it looks like it would enable that functionality.

Also, cross posting can cause us to do extra work as the question may have already been answered on the other post. If you cross post, include a link so we’re all on the same page

Partition MongoDB Atlas Data into Realms

1 Like

Thanks Jay. Yes I did read it. I’ve read it all and that seems to be about accessing the data rather than what gets synced so you’re right, it’s not an answer.

Re cross posting - didn’t realise that was an issue (and not sure why it is) but apologies. I’ve just been trying to get to the bottom of this for weeks and can’t seem to get to an outcome.

Cross posting isn’t an issue but it’s much more efficient when all of the data is in one place for future readers - it’s funnels the energies and eyeballs on that topic.

Let me see if I can clarify a bit - What you’re actually asking is about accessing the data, not setting up a sync…

But if User_A wants to share Document_1 with User_B. I’m struggling with that concept.

Generically imagine a case where you have Tasks app. Some tasks are personal and are either stored locally on the device or sync’d but only that user can access them (it’s tied to their _partitionKey) for example.

Expand on that and and suppose you have tasks that can be shared amongst a group - (sharing Document_1 in your case).

In that situation you would have a groups collection that would map user id’s to group id’s and then realm could determine if that user can access that partitions data. So a partition key for users that can access my tasks would be _partitionKey = “group_id=jay_group”. So essentially each shared document would have that groups partition key “jay_group”. That then leads to the users in that group sync’ing with just that groups tasks (e.g. sharing the document).

There’s several ways of implementing this but using prefixes in the partition keys separates the data into natural groups and realm can parse those prefixes to limit access

The way I look at it is not trying to set up sync’ing - its setting up access to those documents you want to share via appropriate partition keys - the sync then just ‘works’ and sharing documents within groups is ‘easy’

1 Like

Thanks @Jay. Makes sense. Only issue I see is being able to update those groups dynamically where users might be removed and/or added to a document. So effectively being able to update who has access to the _partitionKey = “group_id=jay_group”. That’s the challenging part…

1 Like

That’s kind of the purpose of the group - it can be dynamically changed as you want allow/deny access for users to your data.

So if the user was not in the group (collection), they could not access that groups data.

Just to chip in my 2 cents. As there are limits on what partitions can and can’t filter (e.g. you sync the whole document or none of it), you need to consider your schema as well as your partition key (e.g., in some cases you might need an additional collection and/or duplicate some data using triggers). This article steps through how some reasonably complex partitioning requirements were implemented (together with the though process behind the decisions).

1 Like

Hi @Anthony_CJ, I am having a very similar scenario as yours. How did you end up implementing this? Thanks in advance.

I’m working out a very similar architecture which may end up being implemented via Flexible Sync when we get the greenlight from MongoDB to trust it in production. (I’m hanging out for an announcement at MongoDB World in a month.)

In the meantime, a summary of my thinking:

The context for the app I’m porting is that a small number of appointments and work shifts can be shared with other people. The fan-out is likely to be a max of 8, as a GUI limitation and also social experience.

A limitation that hasn’t come up in this discussion is a restriction on the number of Realms you can have open at the same time. An answer relayed from Andrew Morgan was that 10 was a safe limit, derived from 6-8 file handles per Realm.

There’s no hard-coded limit in Realm, it’s down to the number of file descriptors that various mobile devices allow an app to use (each open Realm uses 6-8). More modern devices allow more, and there are differences between iOS and Android (the lower limits tend to be on iOS).

That limit has informed my thinking - it’s not feasible to have every single tiny shared item in its own Realm (partition).

Without requiring users to define explicit teams, I’m planning to create our own sync pools which contain every person who has agreed to share some stuff with another. Whilst that sounds like it might leak into being a large group, I doubt it in practice.

So a single partition key will be used for each pool. Due to this flood fill approach to defining the pool boundaries, a single user is unlikely to open more than one of them.

Within the pool, access to individual shared items will be further restricted by having an Alice shared with Bob pair of identifiers in a Realm table.

Most of a user’s data will be stored in another individual synced Realm. This provides backups and the chance to have mirroring with a spouse, as a feature.

Initially, people will start completely offline, only when they pay for sharing will they migrate to a Synced Realm.

Pretty sure this is mis-stated. There’s only one partition per open Realm. Your app can have many Realms open (I’ve seen 10 simultaneously as a suggested limit).