Back in November 2020 I enabled GitHub integration with Mongo Atlas, exported the app, linked to GitHub and it’s been working fine. Recently I noticed a change in the file structure when I made changes in the UI. E.g. in GitHub I have files in services\DataCluster\rules to describe the schema rules, yet when I modify those files via the UI it says I have changed a file in data_sources\DataCluster\Database\collection. I was keen to get the files back in sync as it was getting difficult to track the changes, so I exported the App to a zip file again, and copied the files with the new structure into my source and committed GitHub. However, the commit resulted in an error “Failed: failed to import app: cannot link cluster for Atlas service via source control” on the deployment page. I then rolled back to the previous commit but my application failed to link to the Cluster. It took several hours for me to finally get it working again by comparing files in source control individually. I’m now back to the original file structure that I was hoping to move away from. So this doesn’t happen again - how do I get my source control in line with the latest Realm config file structure? The obvious method caused a catastrophic failure.
Hi @ConstantSphere, you’ve hit a glitch in the way that the new export structure is created and imported.
The workaround is to edit the exported app. In the definition of your Atlas connection (typically in data_sources/mongodb-atlas/config.json, remove the config.clusterName attribute — the app should then import correctly.
Hi @Andrew_Morgan. Thanks for that. I did already try removing the entire file but I’ll try removing that line. In my case the file is in data_sources/DataCluster/config.json and looks like:
{ "name": "DataCluster", "type": "mongodb-atlas", "config": { "clusterName": "DataCluster", "readPreference": "primary", "wireProtocolEnabled": false }, "version": 1 }
so you just want me to delete line 5 entirely from that file. I’m being extremely cautious as I don’t want to spend time fixing it if it goes wrong again. I didn’t spot any documentation on the file structures in use (I may have missed it) can you point me in the right direction? Once I’m clear on the file structure I’m hoping to use the new Environment variables feature but again, I haven’t seen anywhere that describes the syntax of what I need to put in my files and where. Right now I am using github directives to stop certain files in my dev branch updating my production branch but that is not ideal.
As a side note, I just looked at my current file which is in services\DataCluster\config.json and it doesn’t have line 5 (and hasn’t been changed since I first set it up in November last year) but is also version 1. I’m wondering if you need to up the version number if the format has changed.
Many thanks for your help.
Hi @ConstantSphere, you should work with the files that have been exported in the new format.
Yes, you should remove the line "clusterName": "DataCluster" from that file.
Hi @Andrew_Morgan. Well that got me a step forward in that importing the new file structure did actually hook up to the cluster but there are still some serious issues.
- all of my schema rules have gone missing - according to the UI there are no schema rules defined
- all of my http services have gone missing - clicking on 3rd party services and http displays a 404 error page:
I’m going to try and roll back to the previous version again.
Edit: I managed to roll back within 20 minutes this time. Also note that the App Users, Authentication methods, Functions and Triggers all seemed to be correct (couldn’t easily test them though as my app wouldn’t function with the missing schema rules).
Many thanks for you help so far, any further suggestions gratefully received.
On advice from support, I tried this again today. Specifically I tried the following…
- MANAGE > Deployment > Export App > Export App as a .zip file
- Replaced the content of my Mongo directory in GitHub with the contents of the zip file
- Synced GitHub with Mongo (via VS Code)
- I got an error in the Deployment status “Failed: No file named metadata.json found in MongoConfig/hosting”
- I then copied the previous contents of the hosting file into source control and tried again
- The deployment appeared to succeed in the Deployment window
- However, once again all of the schema rules and 3rd party services were missing
- I logged out and the back in again to Atlas
- Still the same problem.
- I did not get the issue with the Linked Data Sources failing which now appears to be fixed.
Any further help much appreciated. Thanks!
@Andrew_Morgan - something else seems to have broken around this recently which is a bit more worrying. I recently did a production deployment via GitHub (using the old directory structure which works fine). The deployment to production appeared to work fine in the UI (under deployments) and the release was working fine in my development and staging environments.
However, we noticed a production defect, which after a lot of investigation, was traced back to a schema rule that was using the pre-release behaviour. Upon checking the UI, the rule showed the new code but this was not the code being executed. The solution was to add a single space character to the rule in the UI and deploy the change through the UI.
Normally we would never make any changes via the UI in the production account which makes the release process very fragile. Any help greatly appreciated!
I just posted an update on this on another thread but thought I’d do it here too… About a month ago I built up the courage to try another deployment from GitHub with my “old” directory structure and it mostly worked. By that I mean that the deployment appeared successful in the deployment log, all of the Rules, Webhooks and functions worked and the app remained connected to the cluster. Realm then synced back to GitHub with the “new” directory structure which was fine except that all of my hosted files were missing and therefore my entire website disappeared!
Despite that - this is the best it’s ever worked. If you don’t use the hosting feature this might all be fine. As it happened I could quite easily rebuild the hosting folder via my CI chain so no big harm done. I now have the new directory structure in github and can successfully deploy to github and onto Realm.
However, there is still an issue that I can’t change anything in the Realm UI whist I have “Enable Automatic Deployment” on, otherwise all my hosted files get deleted both in Realm and in my githib repo! This is only really an issue on my dev environment as I would never change anything via the UI in prod.
There is also a feature that I think has been broken in the process… You are supposed to be able to use environment specific variables in the config files so that you don’t have to have different files for each environment. However, when I tried this the placeholder got replaced by the actual value and then synced back to github. I may have been doing it wrong, but if anyone has used this feature successfully with GitHub then please let me know.
I’ve seen the same behavior, values getting replaced when syncing back to Github. It is impossible to use staging and master branches because every pr from staging into master will carry the hardcoded values.
My workaround for the environment variables is to have different config file versions in GitHub for different branches. To stop GitHub overwriting the values of config.json during a merge I use a .gitattributes file in that directory with the contents:
config.json merge=ours
It works but not great if you ever have to merge back from master to a branch then (which due to other Mongo issues I’ve had to do). Using the environment variables is obviously preferred but unfortunately unusable.
Just coming back to this topic as I noticed a change in behaviour today. For the last coupe of years I’ve had no "clusterName": "DataCluster" in the config.json file mentioned above.
When syncing GitHub today I got the error:
Failed
: failed to import app: error validating Service: DataCluster: expected ‘uri’ to be a ‘string’, but instead found a ‘null’
The only way to successfully deploy I could find was to put the line back in. As I required different cluster names for different environments I tried using environment variables e.g. “%%environment.clusterName” but got the error:
Failed
: error processing request
In the end, I reverted back to the workaround mentioned above with the .gitattributes file to have different values in my development and production branches.
Is this a regression? Anyone else seeing this?