Hi,
I’ve been using MongoDB’s GridFS in a sharded cluster setup (3 shards, 1 replica per shard, and version 4.4) and it’s worked impressively well performance-wise so I’d really like to be able to keep using it.
However, I’m running into an issue when it comes to managing the dataset overtime.
The use case is a distributed cache (documents from 300kB to 30MB or so), and the documents in-cache are rolling overtime (as documents are added and others are deleted to stay under disk capacity). Evictions happen when the dataset goes above 500GB, and I can safely pick evicted documents at random.
So, I followed the recommendation at GridFS — MongoDB Manual and am sharding with { files_id : 1, n : 1 }.
However, and it seems to be a known issue, I end up with many empty (or almost empty) chunks after a few runs of deletions, causing my shards to become unbalanced disk-size wise (since I can’t really query for an even spread across shards at the application level).
I noticed mongo/sharding_utils.js at master · vmenajr/mongo (github.com) as seemingly being a recurringly-mentionned solution, but was hoping there could be a better alternative.
This is a bit of a call to the void, but thanks to whoever reads this!
Regards