So if I am reading correctly your current expected size is ~400*500mb=~200gb.
Additionally it sounds like branch is the isolation level where a single branch should not have access to others.
Also you should benefit from tiering branches into different replica set based on their activity/size and/or geolocations this way you can size/place hw and tune queries more specifically.
Since the number of tenants may grow significantly having a collection per tenant in a database per branch might have some scalablity problems as it may potentially result in many files on disk for indexes and collections. Wired Tiger storage might consume more memory and have more open file handles with this approach.
I would think about the following options:
- Within the replica set have a database per branch where the collections inside have a tenant id field indexed and used in queries.
Advantages: better data isolation as roles can be defined on a database level. A better structure if running across different replica set as in each case there will be a database per branch. More structured privilege system relaying on MongoDB roles and permissions.
Disadvantages: the application user will need to have more privileges to dynamically create databases and index new branches on the fly.
If sharding is considered you will need to eventually shard more collections which may result in a less scalable solution. May still have many collections pushing the Wired Tiger to its file limits
- Have the branch and tenant id indexed as a compound index in a main per replica set database and collection for all tenants.
Advantages: less collections which allows less overhead on files. Easier to shard in the future and use sharding as a scale out approach. Will require less administration privs from the application.
Disadvantages Less isolated as application bugs might query wrong branch data. If branches use the data very differently you might endup with locks and lots of indexes on a single collection which might impact performance of writes.
I recommend reviewing the following information: