Didn't get a chance to attend the Realm Kotlin Multiplatform for modern mobile apps Meetup? Don't worry, we recorded the session and you can now watch it at your leisure to get you caught up.
In this meetup, Claus Rørbech, software engineer on the Realm Android team, will walk us through some of the constraints of the RealmJava SDK, the thought process that went into the decision to build a new SDK for Kotlin, the benefits developers will be able to leverage with the new APIs, and how the RealmKotlin SDK will evolve.
In this 50-minute recording, Claus spends about 35 minutes presenting an overview of the Realm Kotlin Multiplatfrom. After this, we have about 15 minutes of live Q&A with Ian, Nabil and our Community. For those of you who prefer to read, below we have a full transcript of the meetup too. As this is verbatim, please excuse any typos or punctuation errors!
Throughout 2021, our Realm Global User Group will be planning many more online events to help developers experience how Realm makes data stunningly easy to work with. So you don't miss out in the future, join our Realm Global Community and you can keep updated with everything we have going on with events, hackathons, office hours, and (virtual) meetups. Stay tuned to find out more in the coming weeks and months.
To learn more, ask questions, leave feedback, or simply connect with other Realm developers, visit our community forums. Come to learn. Stay to connect.
Claus: Yeah, hello. I'm Claus Rorbech, welcome to today's talk on Realm Kotlin. I'm Claus Rorbech and a software engineer at MongoDB working in the Java team and today I'm going to talk about Realm Kotlin and why we decided to build a complete new SDK and I'll go over some of the concepts of this.
We'll do this with a highlight of what Realm is, what triggered this decision of writing a new SDK instead of trying to keep up with the Realm Java. Well go over some central concepts as it has some key significant changes compared to Realm Java. We'll look into status, where we are in the process. We'll look into where and how it can be used and of course peek a bit into the future.
Just to recap, what is Realm? Realm is an object database with the conventional ACID properties. It's implemented in a C++ storage engine and exposed to various language through language specific SDKs. It's easy to use, as you can define your data model directly with language constructs. It's also performant. It utilizes zero copying and lazy loading to keep the memory footprint small. Which is still key for mobile development.
Historically we have been offering live objects, which is a consistent view of data within each iteration of your run loop. And finally we are offering infrastructure for notifications and easy on decide encryption and easy synchronization with MongoDB Realm. So, Realm Java already works for Kotlin on Android, so why bother doing a new SDK? The goal of Realm is to simplify app development. Users want to build apps and not persistent layers, so we need to keep up providing a good developer experience around Realm with this ecosystem.
Why not just keep up with the Realm Java? To understand the challenge of keeping up with Realm Java, we have to have in mind that it has been around for almost a decade. Throughout that period, Android development has changed significantly. Not only has the language changed from Java to Kotlin, but there's also been multiple iterations of design guidelines. Now, finally official Android architectural guidelines and components. We have kept up over the years. We have constantly done API adjustments. Both based on new language features, but also from a lot of community feedback. What users would like us to do. We have tried to accommodate this major new design approach by adding support for the reactive frameworks.
Both RX Java and lately with coroutine flows. But keeping up has become increasingly harder. Not only just by the growing features of Realm itself, but also trying to constantly fit into this widening set of frameworks. We thought it was a good time to reassess some of these key concepts of Realm Java. The fundamentals of Realm Java is built around live objects. They provide a consistent updated view of your data within each run loop iterations. Live data is actually quite nice, but it's also thread confined.
This means that each thread needs its own instance of Realm. This Realm instance needs to be explicitly close to free up resources and all objects obtained from this Realm are also thread confined. These things have been major user obstacles because accessing objects on other threads will flow, and failing to close instances of Realm on background threads can potentially lead to growing file sizes because we cannot clean up this updated data.
So, in general it was awesome for early days Android app architectures that were quite simple, but it's less attractive for the current dominant designs that rely heavily on mutable data streams in frameworks and very flexible execution and threading models. So, besides this wish of trying to redo some of the things, there're other motivations for doing this now. Namely, being Kotlin first. Android has already moved there and almost all other users are also there.
We want to take advantage of being able to provide the cleaner APIs with nice language features like null safety and in line and extension functions and also, very importantly, co routines for asynchronous operations. Another key feature of Kotlin is the Kotlin Compiler plugin mechanism. This is a very powerful mechanism that can substitute our current pre processor approach and byte manipulation approach.
So, instead of generating code along the user classes, we can do in place code transformation. This reduces the amount of code we need to generate and simplifies our code weaving approach and therefore also our internal APIs. The Kotlin Compiler plugin is also faster than our current approach because we can eliminate the very slow KAPT plugin that acts as an annotation processor, but does it by requiring multiple passes to first generate stops and then the actual code for the implementation.
For example, UI. We think Realm fits well into this Kotlin multi platform library suite. There's often no need for developer platform differentiation for persistence and there's actually already quite good coverage in this library suite. Together with the Kotlin serialization and Realm, you can actually do full blown apps as shared code and only supply a platform specific UI.
So, let's look into some of the key concepts of this new SDK. We'll do that by comparing it to Realm Java and we'll start by defining a data model. Throughout all these code snippets I've used the Kotlin syntax even for the Realm Java examples just to highlight the semantic changes instead of bothering with the syntactical differences. So, I just need some water...
So, as you see it looks more or less like Java. But there are some key things there. The compiler plugins enable us access the classes directly. This means we no longer need the classes to be open, because we are not internally inheriting from the user classes. We can also just add a marker interface that we fill out by the compiler plugin instead of requiring some specific base classes. And we can derive nullability directly from the types in Kotlin instead of requiring this required annotation.
Not all migration are as easy to cut down. For our Realm configurations, we're not opting in for pure Kotlin with the named parameters, but instead keeping the binder approach. This is because it's easier to discover from inside the ID and we also need to have in mind that we need to be interoperable with the Java. We only offer some specific constructors with named parameters for very common use cases. Another challenge from this new tooling is that the compiler plug-in has some constraints that complicates gathering default schemas.
We're not fully in place with the constraints for this yet, so for now, we're just required explicit schema definition in the configuration. For completion, I'll just also highlight the current API for perform inquiries on the realm. To get immediate full query capabilities with just exposed the string-based parcel of the underlying storage engine, this was a quick way to get full capabilities and we'll probably add a type safe query API later on when there's a bigger demand.
Actually, this string-based query parcel is also available from on Java recently, but users are probably more familiar with the type-based or type safe query system. All these changes are mostly syntactical but the most dominant change for realm Kotlin is the new object behavior. In Realm Kotlin, objects are no longer live, but frozen. Frozen objects are data objects tied to a specific version of the realm. They are immutable. You cannot update them and they don't change over time.
We still use the underlying zero-copying and lazy loading mechanism, so we still keep the memory footprints small. You can still use a frozen object to navigate the full object graph from this specific version. In Realm Kotlin, the queries also just returns frozen objects. Similarly, notifications also returns new instances of frozen objects, and with this, we have been able to lift the thread confinement constraint. This eases lifecycle management, because we can now only have a single global instance of the realm.
We can also pass these objects around between threads, which makes it way easier to use it in reactive frameworks. Again, let's look into some examples by comparing is to Realm Java. The shareable Realm instances eases this life cycle management. In Realm Java, we need to obtain an instance on each thread, and we also need to explicitly close this realm instance. On Realm Kotlin, we can now do this upfront with a global instance that can be passed around between the threads. We can finally close it later on this single instance. Of course, it has some consequences. With these shareable instances, changes are immediately available too on the threads.
In Realm Java, the live data implementation only updated our view of data with in between our run loop iterations. Same query in the same scope would always yield the same result. For Realm Kotlin with our frozen objects, so in Realm Kotlin, updates from different threads are immediately visible. This means that two consecutive queries might reveal different results. This also applies for local or blocking updates. Again, Realm Java with live results local updates were instantly visible, and didn't require refresh. For Realm Java, the original object is frozen and tied to a specific version of the Realm.
Which means that the update weren't reflected in our original object, and to actually inspect the updates, we would have to re-query the Realm again. In practice, we don't expect access to these different versions to be an issue. Because the primary way of reacting to changes in Realm Kotlin will be listening for changes.
In Realm Kotlin, updates are delivered as streams of immutable objects. It's implemented by coroutine flows of these frozen instances. In Realm Java, when you got notified about changes, you could basically throw away the notification object, because you could still access all data from your old existing live reference. With Realm Kotlin, your original instance is tied to a specific version of the Realm. It means that you for each update, you would need to access the notify or the new instance supplied by the coroutine flow.
But again, this updated instance, it gives you full access to the Realm version of that object. It gives you full access to the object graph from this new frozen instance. Here we start to see some of the advantage of coroutine based API. With Realm Java, this code snippet, it was run on a non-loop of thread. It would actually not even give you any notification because it was hard to tie the user code with our underlying notification mechanism. For Realm Kotlin, since we're using coroutines, we use the flexibilities of this. This gives the user flexibility to supply this loop like dispatching context and it's easy for us to hook our event delivery off with that.
Secondly, we can also now spread the operations on a flow over varying contexts. This means that we can apply all the usual flow operations, but we can also actually change the context as our objects are not tied to a specific thread. Further, we can also with the structural concurrency of codes routines, it's easier to align our subscription with the existing scopes. This means that you don't have to bother with closing the underlying Realm instance here.
With the shareable Realm instances of Realm Kotlin, updates must be done atomically. With Realm Java, since we had a thread confined instance of the Realm, we could just modify it. In Realm Kotlin, we have to be very explicit on when the state is changed. We therefore provide this right transaction on a separate mutable Realm within a managed scope. Inside the scope, it allows us to create an update objects just as in Realm Java. But the changes are only visible to other when once the scope is exited. We actually had a similiar construct in Realm Java, but this is now the only way to update the Realm.
Inside the transaction blocks, things almost works as in Realm Java. It's the underlying same starch engine principles. Transactions are still performed on single thread confined live Realm, which means that inside this transaction block, the objects and queries are actually live. The major key difference for the transaction block is how to update existing objects when they are now frozen. For both STKs, it applies that we can only do one transaction at a time. This transaction must be done on the latest version of the Realm. Therefore, when updating existing objects, we need to ensure that we have an instance that is tied to the latest version of the object.
For Realm Java, we could just pass in the live objects to our transaction block but since it could actually have ... we always had to check the object for its validity. A key issue here was that since object couldn't be passed around on arbitrary threads, we had to get a non-local object, we would have to query the Realm and find out a good filtering pattern to identify objects uniquely. For Realm, we've just provided API for obtaining the latest version of an object. This works for both primary key and non-primary key objects due to some internal identifiers.
Since we can now pass objects between a thread, we can just pass our frozen objects in and look up the latest version. To complete the tour, we'll just close the Realm. As you've already seen, it's just easier to manage the life cycle of Realm when there's one single instance, so closing an instance to free up resources and perform exclusive operations on the Realm, it's just a matter of closing the shared global instance.
Interacting with any object instance after you closed the Realm will still flow, but again, the structural concurrency of coroutine flows should assist you in stopping accessing the objects following the use cases of your app. Besides the major shift to frozen objects, we're of course trying to improve the STKs in a lot of ways, and first of all, we're trying to be idiomatic Kotlin. We want to take advantage of all the new features of the language. We're also trying to reduce size both of our own library but also the generated code. This is possible with the new compiler plug-in as we've previously touched. We can just modify the user instance and not generate additional classes.
We're also trying to bundle up part of functionality and modularize it into support libraries. This is way easier with the extension methods. Now we should be able to avoid having everybody to ship apps with the JSON import and export functionality and stuff like that. This also makes it easier to target future frameworks by offering support libraries with extension functions. As we've already also seen, we are trying to ensure that our STK is as discoverable from the ID directly, and we're also trying to ensure that the API is backward compatible with the Java.
This might not be the most idiomatic Java, but at least we try to do it without causing major headaches. We also want to improve testability, so we're putting in places to inject dispatchers. But most notably, we're also supplying JBM support for off-device testing. Lastly, since we're redoing an STK, we of course have a lot of insight in the full feature set, so we also know what to target to make a more maintainable STK. You've already seen some of the compromises for these flights, but please feel free to provide feedback if we can improve something.
With this new multiplatform STK, where and how to use. We're providing our plug-in and library as a Kotlin multiplatform STK just to be absolutely clear for people not familiar with the multiplatform ecosystem. This still means that you can just apply your project or apply this library and plug-in on Android only projects. It just means that we can now also target iOS and especially multiplatform and later desktop JBM. And I said there's already libraries out there for sterilization and networking.
With Realm, we can already build full apps with the shared business logic, and only have to supply platform dependent UI. Thanks to the layout and metadata of our artifacts, there's actually no difference in how to apply the projects depending on which setting you are in. It integrates seamlessly with both Android and KMM projects. You just have to apply the plug-in. It's already available in plug-in portal, so you can just use this new plug-in syntax. Then you have to add the repository, but you most likely already have Maven Central as part of your setup, and then at our dependency. There's a small caveat for Android projects before Kotlin or using Kotlin, before 1.5, because the IR backend that triggers our compiler plug-in is not default before that.
You would have to enable this feature in the compiler also. Yeah. You've already seen a model definition that's a very tiny one. With this ability or this new ability to share our Realm instances, we can now supply one central instance and here have exemplified it by using a tiny coin module. We are able to share this instance throughout the app, and to show how it's all tied together, I have a very small view model example. These users are central Realm instance supplied by Kotlin.
It sets up some live data to feed the view. This live data is built up from our observable flows. You can apply the various flow operators on it, but most importantly you can also control this context for where it's executing. Lastly, you are handling this in the view model scope. It's just that subscription is also following the life cycle of your view model. Lastly, for completion, there's a tiny method to put some data in there.
As you might have read between the lines, this is not all in place yet, but I'll try to give a status. We're in the middle of maturing this prove of concept of the proposed frozen architecture. This is merged into our master branch bit by bit without exposing it to the public API. There's a lot of pieces that needs to fit together before we can trigger and migrate completely to this new architecture. But you can still try our library out. We have an initial developer preview in what we can version 0.1.0. Maybe the best label is Realm Kotlin Multiplatform bring-up.
Because it sort of qualifies the overall concept in this mutliplatform setting with our compiler plug-in being on multi platforms. Also a mutliplatform [inaudible 00:30:09] with collecting all these native objects in the various [inaudible 00:30:14] management domains. A set, it doesn't include the full frozen architecture yet, so the Realm instances are still thread confined, and objects are live. There's only limited support, but we have primitive types, links to other Realm objects and primary keys. You can also register for notifications.
We use this string-based queries also briefly mentioned. It operates on Kotlin Mutliplatform mobile, which means that it's both available for Android and iOS, but only for 64 bit on both platforms. It's already available on Maven Central, so you can go and try it out either by using our own KMM example in the repository or build your own project following the read me in the repository.
What's next? Yeah. Of course in these weeks, we're stabilizing the full frozen architecture. Our upcoming milestones are first we want to target the release with the major Java features. It's a lot of the other features of Realm like lists, indexes, more detailed changelistener APIs, migration for schema updates and dynamic realms and also desktop JVM support. After that, we'll head off to build support for MongoDB Realm. To be able to sync this data remotely.
When this is in place, we'll target the full feature set of Realm Java. There's a lot of more exotic types embedded objects and there's we also just introduced new types like sets and dictionaries and [inaudible 00:32:27] types to Realm Java. These will come in a later version on Kotlin. We're also following the evolution of Kotlin Mutliplatform. It's still only alpha so we have to keep track of what they're doing there, and most notably, following the memory management model of Kotlin native, there are constraints that once you pass objects around, Kotlin is freezing those.
Right now, you cannot just pass Realm instances around because they have to be updated. But these frozen objects can be passed around threads and throughout this process, we'll do incremental releases, so please keep an eye open and provide feedback.
To keep up with our progress, follow us on GitHub. This is our main communication channel with the community. You can try out the sample, a set. You can also ... there's instructions how to do your own Kotlin Mutliplatform project, and you can peek into our public design docs. They're also linked from our repository. If you're more interested into the details of building this Mutliplatform STK, you can read a blog post on how we've addressed some of this challenge with the compiler plug-in and handling Mutliplatform C Interrupts, memory management, and all this.
Thank you for ... that's all.
Ian: Thank you Claus, that was very enlightening. Now, we'll take some of your questions, so if you have any questions, please put them in the chat. The first one here will mention, we've answered some of them, but the first one here is regarding the availability of the. It is available now. You can go to GitHub/Realm/Realm.Kotlin, and get our developer preview. We plan to have iterative releases over the next few quarters. That will add more and more functionality.
The next one is regarding the migration from I presume this user has ... or James, you have a Realm Java application using Realm Java and potentially, you would be looking to migrate to Realm Kotlin. We don't plan to have an automatic feature that would scan your code and change the APIs. Because the underlying semantics have changed so much. But it is something that we can look to have a migration guide or something like that if more users are asking about it.
Really the objects have changed from being live objects to now being frozen objects. We've also removed the threading constraint and we've also have a single shared Realm instance. Whereas before, with every thread, you had to open up a new Realm instance in order to do work on that thread. The semantics have definitely changed, so you'll have to do this with developer care in order to migrate your applications over. Okay. Next question here, we'll go through some of these.
Does the Kotlin STK support just KMM for syncing or just local operations? I can answer this one. We do plan to offer our sync, and so if you're not familiar, Realm also offers a synchronization from the local store Realm file to MongoDB Atlas through Realm Sync, through the Realm Cloud. This is a way to bidirectionally sync any documents that you have stored on MongoDB Atlas down and transformed into Realm objects and vice versa.
We don't have that today, but it is something that you can look forward to the future in next quarters, we will be releasing our sync support for the new Realm Kotlin STK. Other questions here, so are these transactions required to be scoped or suspended? I presume this is using the annotations for the Kotlin coroutines keywords. The suspend functions, the functions, Claus, do you have any thoughts on that one?
Claus: Yeah. We are providing a default mechanism but we are also probably adding at least already in our current prototype, we already have a blocking right. You will be able to do it without suspending. Yeah.
Ian: Okay. Perfect. Also, in the same vein, when running a right transaction, do you get any success or failed result back in the code if the transaction was successful? I presume this is having some sort of callback or on success or on failure if the right transaction succeeded or failed. We plan that to our API at all?
Claus: Usually, we just have exceptions if things doesn't go right, and they will propagate throughout normal suspend ... throughout coroutine mechanisms. Yeah.
Ian: Yeah. It is a good thought though. We have had other users request this, so it's something if we continue to get more user feedback on this, potentially we could add in the future. Another question here, is it possible to specify a path to where all the entities are loaded instead of declaring each on a set method?
Ian: Not sure I fully follow that.
Claus: It's when defining the schema. Yeah. We have some options of gathering this schema information, but as I stated, we are not completely on top of which constraints we want to put into it. Right now, we are forced to actually define all the classes, but we have issues for addressing this, and we have investigating various options. But some of these come with different constraints, and we have to judge them along the way to see which fits best or maybe we'll find some other ways around this hopefully.
Nabil: Just to add on top of this, we could use listOf or other data structure. Just the only constraint at the compiler level are using class literal to specify the. Since there's no reflection in Kotlin Native, we don't have the mechanism like we do in Java to infer from your class path, the class that are annotating to Java that will participate in your schema. The lack of reflection in Kotlin Native forces us to just use class and use some compiler classes to build the schema for you constraint.
Ian: Yeah. Just to introduce everyone, this is Nabil. Nabil also works on the Realm Android team. He's one of the lead architects and designers of our new Realm Kotlin STK, so thank you Nabil.
Nabil: Sorry. I lost you for a bit.
Claus: I also lost the initial. No, but we are supporting the full feature set, so I can't immediately come up with any constraints around Java.
Ian: Other questions here, will the Realm SDK support other platforms not just mobile? We talked a little bit about desktop, so I think JVM is something that we think we can get out of the box with our implementations of course. I think for desktop JVM applications is possible.
Nabil: Internally like I mentioned already compiling for JVM and and also for. But we didn't expose it other public API yet. We just wanted object support in iOS and Android. The only issue for JVM, we have the tool chain compiling on desktop file, is to add the Android specific component, which are like the which you don't have for desktops. We need to find way either to integrate with Swing or to provide a hook for you to provide your looper, so it can deliver a notification for you. That's the only constraint since we're not using any other major Android specific API besides the context. The next target that we'll try to support is JVM, but we're already supported for internally, so it's not going to be a big issue.
Ian: I guess in terms of web, we do have ... Realm Core, the core database is written in C++. We do have some projects to explore what it would take to compile Realm Core into Wasm so that it could then be run into a browser, and if that is successful, then we could potentially look to have web as a target. But currently, it's not part of our target right now.
Other questions here, will objects still be proxies or will they now be the same object at runtime? For example, the object Realm proxy.
Nabil: There will be no proxy for Realm Kotlin. It's one of the benefit of using a compiler, is we're modifying the object itself. Similar to what composes doing with the add compose when you write a compose UI. It's modifying your function by adding some behavior. We do the same thing. Similar to what Kotlin sterilization compiler is doing, so we don't use proxy objects anymore.
Ian: Okay. Perfect. And then I heard at the end that Realm instances can't be frozen on Native, just wanted to confirm that. Will that throw off if a freeze happens? Also wondering if that's changing or if you're waiting on the Native memory model changes.
Nabil: There's two aspects to what we're observing to what are doing. It's like first it's the garbage collector. They introduce an approach, where you could have some callbacks when you finalize the objects and we'll rely on this to free the native resources. By native, I mean the C++ pointer. The other aspect of this is the memory model itself of the Kotlin Native, which based on frozen object similiar concept as what we used to do in Realm Java, which is a thread confinement model to achieve.
Nabil: Actually, what we're doing is like we're trying to freeze the object graph similarly to what Kotlin it does. You can pass object between threads. The only sometimes issue you is like how you interface with multi-threaded coroutine on Kotlin Native. This part is not I think stable yet. But in theory, our should overlap, should work in a similiar way.
Claus: I guess we don't really expect this memory management scheme from Kotlin Native to be instantly solved, but we have maybe options of providing the Realm instance in like one specific instance that doesn't need to be closed on each thread, acting centrally with if our Native writer and notification thread. It might be possible in the future to define Realms on each thread and interact with the central mechanisms. But I wouldn't expect the memory management constraints on Native to just go away.
Ian: Right. Other question here will Realm plan on supporting the Android Paging 3 API?
Nabil: You could ask the same question with Realm Java. The actual lazy loading capability of Realm doesn't require you to implement the paging library. The paging library problem is like how you can load efficiently pages of data without loading the entire dataset. Since both Realm Java and Realm Kotlin uses just native pointers to the data, so as you traverse your list or collection will only loads this object you're trying to access. And not like 100 or 200 similiar to what a cursor does in SQLite for instance. There's no similiar problem in Realm in general, so it didn't try to come up with a paging solution in the first place.
Ian: That's just from our lazy loading and memory map architecture, right?
Ian: Yeah. Okay. Other question here, are there any plans to support polymorphic objects in the Kotlin SDK? I can answer this. I have just finished a product description of adding inheritance in polymorphism to not only the Kotlin SDK, but to all of our SDKs. This is targeted to be a medium term task. It is an expensive task, but it is something that has been highly requested for a while now. Now, we have the resources to implement it, so I'd expect we would get started in the medium term to implement that.
Nabil: We also have released for in Java what we call the polymorphic type, which is. You can and then install in it the supported Realm have JSON with the dynamic types, et cetera. Go look at it. But it's not like the polymorphic, a different polymorphic that Ian was referring -
Ian: It's a first step I guess you could say into polymorphism. What it enables is kind of what Nabil described is you could have an owner field and let's say that owner could be a business type, it could be a person, it could be an industrial, a commercial, each of these are different classes. You now have the ability to store that type as part of that field. But it doesn't allow for true inheritance, which I think is what most people are looking for for polymorphism. That's something that is after this is approved going to be underway. Look forward to that. Other questions here? Any other questions? Anything I missed? I think we've gone through all of them. Thank you to the Android team. Here's Christian, the lead of our Android team on as well. He's been answering a lot of questions. Thank you Christian. But if any other questions here, please reach out. Otherwise, we will close a little bit early.
Okay. Well, thank you so much everyone. Claus, thank you. I really appreciate it. Thank you for putting this together. This will be posted on YouTube. Any other questions, please go to /Realm/Realm.Kotlin on our GitHub. You can file an issue there. You can ask questions. There's also Forums.Realm.io.
We look forward to hearing from you. Okay. Thanks everyone. Bye.
Nabil: See you online. Cheers.