HomeLearnHow-to

Realm Data and Partitioning Strategy Behind the WildAid O-FISH Mobile Apps

Published: Mar 26, 2021

  • Realm
  • MongoDB
  • Mobile
  • ...

By Andrew Morgan

Share

In 2020, MongoDB partnered with the WildAid Marine Protection Program to create a mobile app for officers to use while out at sea patrolling Marine Protected Areas (MPAs) worldwide. We implemented apps for iOS, Android, and web, where they all share the same Realm back end, schema, and sync strategy. This article explains the data architecture, schema, and partitioning strategy we used. If you're developing a mobile app with Realm, this post will help you design and implement your data architecture.

MPAs—like national parks on land—set aside dedicated coastal and marine environments for conservation. WildAid helps enable local agencies to protect their MPAs. We developed the O-FISH app for enforcement officers to search and create boarding reports while they're out at sea, patrolling the MPAs and boarding vessels for inspection.

O-FISH needed to be a true offline-first application as officers will typically be without network access when they're searching and creating boarding reports. It's a perfect use case for the Realm mobile database and MongoDB Realm Sync.

This video gives a great overview of the WildAid Marine Program, the requirements for the O-FISH app, and the technologies behind the app:

This article is broken down into these sections:

#The O-FISH Application

There are three frontend applications.

The two mobile apps (iOS and Android) provide the same functionality. An officer logs in and can search existing boarding reports, for example, to check on past reports for a vessel before boarding it. After boarding the boat, the officer uses the app to create a new boarding report. The report contains information about the vessel, equipment, crew, catch, and any laws they're violating.

Crucially, the mobile apps need to allow users to view and create reports even when there is no network coverage (which is the norm while at sea). Data is synchronized with other app instances and the backend database when it regains network access.

Animation showing the iOS O-FISH application being used to create a boarding report. A new report is created, details are added using different tabs before the new report is submitted.
iOS O-FISH App in Action

The web app also allows reports to be viewed and edited. It provides dashboards to visualize the data, including plotting boardings on a map. User accounts are created and managed through the web app.

All three frontend apps share a common backend Realm application. The Realm app is responsible for authenticating users, controlling what data gets synced to each mobile app instance, and persisting the data to MongoDB Atlas. Multiple "agencies" share the same frontend and backend apps. An officer should have access to the reports belonging to their agency. An agency is an authority responsible for enforcing the rules for one or more regional MPAs. Agencies are often named after the country they operate in. Examples of agencies would be Galapogas or Tanzania.

#System Architecture

The iOS and Android mobile apps both contain an embedded Realm mobile database. The app reads and writes data to that Realm database-whether the device is connected to the network or not. Whenever the device has network coverage, Realm synchronizes the data with other devices via the Realm backend service.

System architecture for the O-FISH apps. The backend Realm app sits in the middle, synchronizing data between the Realm mobile database embedded in both the iOS and Android apps. Realm also persists the data to MongoDB Atlas and makes that data available to the web app through the Realm SDK.
O-FISH System Architecture

The Realm database is embedded within the mobile apps, each instance storing a partition of the O-FISH data. We also need a consolidated view of all of the data that the O-FISH web app can access, and we use MongoDB Atlas for that. MongoDB Realm is also responsible for synchronizing the data with the MongoDB Atlas database.

The web app is stateless, accessing data from Atlas as needed via the Realm SDK.

MongoDB Charts dashboards are embedded in the web app to provide richer, aggregated views of the data.

#Data Partitioning

MongoDB Realm Sync uses partitions to control what data it syncs to instances of a mobile app. You typically partition data to limit the amount of space used on the device and prevent users from accessing information they're not entitled to see or change.

When a mobile app opens a synced Realm, it can provide a partition value to specify what data should be synced to the device.

As a developer, you must specify an attribute to use as your partition key. The rules for partition keys have some restrictions:

  • All synced collections use the same attribute name and type for the partition key.
  • The key can be a string, objectId, or a long.
  • When the app provides a partition key, only documents that have an exact match will be synced. For example, the app can't specify a set or range of partition key values.

A common use case would be to use a string named "username" as the partition key. The mobile app would then open a Realm by setting the partition to the current user's name, ensuring that the user's data is available (but no data for other users).

If you want to see an example of creating a sophisticated partitioning strategy, then Building a Mobile Chat App Using Realm – Data Architecture describes RChat's approach (RChat is a reference mobile chat app built on Realm and MongoDB Realm). O-FISH's method is straightforward in comparison.

WildAid works with different agencies around the world. Each officer within an agency needs access to any boarding report created by other officers in the same agency. Photos added to the app by one officer should be visible to the other officers. Officers should be offered menu options tailored to their agency—an agency operating in the North Sea would want cod to be in the list of selectable species, but including clownfish would clutter the menu.

We use a string attribute named agency as the partitioning key to meet those requirements.

As an extra level of security, we want to ensure that an app doesn't open a Realm for the wrong partition. This could result from a coding error or because someone hacks a version of the app. When enabling Realm Sync, we can provide expressions to define whether the requesting user should be able to access a partition or not.

Using the Realm UI to define expressions for the read and write permissions to control whether the current user can access a partition or not.
Expression to Limit Sync Access to Partitions

For O-FISH, the rule is straightforward. We compare the logged-in user's agency name with the partition they're requesting to access. The Realm will be synced if and only if they're the same:

1{
2 "%%user.custom_data.agency.name": "%%partition"
3}

#Data Schema

At the highest level, the O-FISH schema is straightforward with four Realms (each with an associated MongoDB Atlas collection):

  • DutyChange records an officer going on-duty or back off-duty.
  • Report contains all of the details associated with the inspection of a vessel.
  • Photo represents a picture (either of one of the users or a photo that was taken to attach to a boarding report).
  • MenuData contains the agency-specific values that officers can select through the app's menus.
UML diagram showing the structure of each of the top-level objects as well as their embedded objects.

You might want to right-click that diagram so that you can open it in a new tab!

Let's take a look at each of those four objects.

#DutyChange

The app creates a DutyChange object when a user toggles a switch to flag that they are going on or off duty (at sea or not).

Abimation of iOS O-FISH app where the user toggles a switch to indicate that they're starting their shift and going on duty.

These are the Swift and Kotlin versions of the DutyChange class:

1import RealmSwift
2
3class DutyChange: Object {
4 @objc dynamic var _id: ObjectId = ObjectId.generate()
5 @objc dynamic var user: User? = User()
6 @objc dynamic var date = Date()
7 @objc dynamic var status = ""
8
9 override static func primaryKey() -> String? {
10 return "_id"
11 }
12}

On iOS, DutyChange inherits from the Realm Object class, and the attributes need to be made accessible to the Realm SDK by making them dynamic and adding the @objc annotation. The Kotlin app uses the @RealmClass annotation and inheritance from RealmObject.

Note that there is no need to include the partition key as an attribute.

In addition to primitive attributes, DutyChange contains user which is of type User:

1import RealmSwift
2
3class User: EmbeddedObject, ObservableObject {
4 @objc dynamic var name: Name? = Name()
5 @objc dynamic var email = ""
6}

User objects are always embedded in higher-level objects rather than being top-level Realm objects. So, the class inherits from EmbeddedObject rather than Object in Swift. The Kotlin app extends the @RealmClass annotation to include (embedded = true).

Whether created in the iOS or Android app, the DutyChange object is synced to MongoDB Atlas as a single DutyChange document that contains a user sub-document:

1{
2 "_id" : ObjectId("6059c9859a545bbceeb9e881"),
3 "agency" : "Ecuadorian Galapagos",
4 "date" : ISODate("2021-03-23T10:57:09.777Z"),
5 "status" : "At Sea",
6 "user" : {
7 "email" : "global-admin@clusterdb.com",
8 "name" : {
9 "first" : "Global",
10 "last" : "Admin"
11 }
12 }
13}

There's a Realm schema associated with each collection that's synced with Realm Sync. The schema can be viewed and managed through the Realm UI:

JSON document describing the schema for the DutyChange collection

#Report

The Report object is at the heart of the O-FISH app. A report is what an officer reviews for relevant data before boarding a boat. A report is where the officer records all of the details when they've boarded a vessel for an inspection.

In spite of appearances, it pretty straightforward. It looks complex because there's a lot of information that an officer may need to include in their report.

Starting with the top-level object - Report:

1import RealmSwift
2
3class Report: Object, Identifiable {
4 @objc dynamic var _id: ObjectId = ObjectId.generate()
5 let draft = RealmOptional<Bool>()
6 @objc dynamic var reportingOfficer: User? = User()
7 @objc dynamic var timestamp = NSDate()
8 let location = List<Double>()
9 @objc dynamic var date: NSDate? = NSDate()
10 @objc dynamic var vessel: Boat? = Boat()
11 @objc dynamic var captain: CrewMember? = CrewMember()
12 let crew = List<CrewMember>()
13 let notes = List<AnnotatedNote>()
14 @objc dynamic var inspection: Inspection? = Inspection()
15
16 override static func primaryKey() -> String? {
17 return "_id"
18 }
19}

The Report class contains Realm List s (RealmList in Kotlin) to store lists of instances of classes such as CrewMember.

Some of the classes embedded in Report contain further embedded classes. There are 19 classes in total that make up a Report. You can view all of the component classes in the iOS and Android repos.

Once synced to Atlas, the Report is represented as a single BoardingReports document (the name change is part of the schema definition):

Single document (containing sub-documents and arrays) from the BoardingReports Atlas collections

Note that Realm lists are mapped to JSON/BSON arrays.

#Photo

A single boarding report could contain many large photographs, and so we don't want to embed those within the Report object (as an object could grow very large and even exceed MongoDB's 16 MB document limit). Instead, the Report object (and its embedded objects) store references to Photo objects. Each photo is represented by a top-level Photo Realm object. As an example, Attachments contains a Realm List of strings, each of which identifies a Photo object. Handling images will step through how we implemented this.

1class Attachments: EmbeddedObject, ObservableObject {
2 let notes = List<String>()
3 let photoIDs = List<String>()
4}

The general rule is that it isn't the best practice to store images in a database as they consume a lot of valuable storage space. A typical solution is to keep the image in some store with plentiful, cheap capacity (e.g., a block store such as Amazon S3.) The O-FISH app's issue is that it's probable that the officer's phone has no internet connectivity when they create the boarding report and attach photos, so uploading them to S3 can't be done at that time. As a compromise, O-FISH stores the image in the Photo object, but when the device has internet access, that image is uploaded to S3, removed from the Photo object and replaced with the S3 link. This is why the Photo includes both an optional binary picture attribute and a pictureURL field for the S3 link:

1class Photo: Object {
2 @objc dynamic var _id: ObjectId = ObjectId.generate()
3 @objc dynamic var thumbNail: NSData?
4 @objc dynamic var picture: NSData?
5 @objc dynamic var pictureURL = ""
6 @objc dynamic var referencingReportID = ""
7 @objc dynamic var date = NSDate()
8
9 override static func primaryKey() -> String? {
10 return "_id"
11 }
12}

Note that we include the referencingReportID attribute to make it easy to delete all Photo objects associated with a Report.

The officer also needs to review past boarding reports (and attached photos), and so the Photo object also includes a thumbnail image for off-line use.

Each agency needs the ability to customize what options are added in the app's menus. For example, agencies operating in different countries will need to define the list of locally applicable laws. Each agency has a MenuData instance with a list of strings for each of the customizable menus:

1class MenuData: Object {
2 @objc dynamic var _id = ObjectId.generate()
3 let countryPickerPriorityList = List<String>()
4 let ports = List<String>()
5 let fisheries = List<String>()
6 let species = List<String>()
7 let emsTypes = List<String>()
8 let activities = List<String>()
9 let gear = List<String>()
10 let violationCodes = List<String>()
11 let violationDescriptions = List<String>()
12
13 override static func primaryKey() -> String? {
14 return "_id"
15 }
16}

#Handling images

When MongoDB Realm Sync writes a new Photo document to Atlas, it contains the full-sized image in the picture attribute. It consumes space that we want to free up by moving that image to Amazon S3 and storing the resulting S3 location in pictureURL. Those changes are then synced back to the mobile apps, which can then decide how to get an image to render using this algorithm:

  1. If picture contains an image, use it.
  2. Else, if pictureURL is set and the device is connected to the internet, then fetch the image from S3 and use the returned image.
  3. Else, use the thumbNail.
Animation showing how an image is synced from the mobile app to Atlas. When it hits Atlas, a database trigger fires and stores the image in S3. The trigger then removes the image from the document and replaces it with the S3 URL. Those changes are then synced to the mobile apps to free up space.

When the Photo document is written to Atlas, the newPhoto database trigger fires, which invokes a function named newPhoto function.

Creating a ``newPhoto`` database trigger through the Realm UI

The trigger passes the newPhoto Realm function the changeEvent, which contains the new Photo document. The function invokes the uploadImageToS3 Realm function and then updates the Photo document by removing the image and setting the URL:

1exports = function(changeEvent){
2const fullDocument = changeEvent.fullDocument;
3const image = fullDocument.picture;
4const agency = fullDocument.agency;
5const id = fullDocument._id;
6const imageName = `${id}`;
7
8if (typeof image !== 'undefined') {
9 console.log(`Requesting upload of image: ${imageName}`);
10 context.functions.execute("uploadImageToS3", imageName, image)
11 .then (() => {
12 console.log('Uploaded to S3');
13 const bucketName = context.values.get("photoBucket");
14 const imageLink = `https://${bucketName}.s3.amazonaws.com/${imageName}`;
15 const collection = context.services.get('mongodb-atlas').db("wildaid").collection("Photo");
16 collection.updateOne({"_id": fullDocument._id}, {$set: {"pictureURL": imageLink}, $unset: {picture: null}});
17 },
18 (error) => {
19 console.error(`Failed to upload image to S3: ${error}`);
20 });
21} else {
22 console.log("No new photo to upload this time");
23}
24};

uploadImageToS3 uses Realm's AWS integration to upload the image:

1exports = function(name, image) {
2 const s3 = context.services.get('AWS').s3(context.values.get("awsRegion"));
3 const bucket = context.values.get("photoBucket");
4 console.log(`Bucket: ${bucket}`);
5 return s3.PutObject({
6 "Bucket": bucket,
7 "Key": name,
8 "ACL": "public-read",
9 "ContentType": "image/jpeg",
10 "Body": image
11 });
12};

#Summary

We've covered the common data model used across the iOS, Android, and backend Realm apps. (The web app also uses it, but that's beyond the scope of this article.)

The data model is deceptively simple. There's a lot of nested information that can be captured in each boarding report, resulting in 20+ classes, but there are only four top-level classes in the app, with the rest accounted for by embedding. The only other type of relationship is the references to instances of the Photo class from other classes (required to prevent the Report objects from growing too large).

The partitioning strategy is straightforward. Partitioning for every class is based on the name of the user's agency. That pattern is going to appear in many apps—just substitute "agency" with "department," "team," "user," "country," ...

Suppose you determine that your app needs a different partitioning strategy for different classes. In that case, you can implement a more sophisticated partitioning strategy by encoding a key-value pair in a string partition key.

For example, if we'd wanted to partition the reports by username (each officer can only access reports they created) and the menu items by agency, then you could partition on a string attribute named partition. For the Report objects, it would be set to pairs such as partition = "user=bill@some-domain.com" whereas for a MenuData object it might be set to partition = "agency=Galapagos". Building a Mobile Chat App Using Realm – Data Architecture steps through designing these more sophisticated strategies.

#Resources

If you have questions, please head to our developer community website where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

MongoDB Icon
  • Developer Hub
  • Documentation
  • University
  • Community Forums

© MongoDB, Inc.