What is the correct way to add a secondary member to an existing shard replica set? I don’t see any documentation about this. Does the shard have to be removed and re-added to the cluster? Would I be able to clone the primary shard and stand up a secondary using that image?
The procedures for adding and removing members from shard replica sets are the same as working with a standalone replica set. Changes in shard replica set configuration or availability are automatically discovered by clients/drivers as part of MongoDB’s distributed database design.
For a guide, see: Add Members to a Replica Set. This tutorial is for the latest production release series of the MongoDB server (currently version 4.4), but versions for older releases can be selected in the navigation on the left column of that docs page. In general I would make sure you are referring to documentation matching your server version, as there may be differences in procedures or available options.
Definitely not. The process of removing a shard will migrate all data from the shard being removed to existing shards, so you would only want to use that approach when decommissioning a shard.
Yes, you can use a valid file copy backup of an existing replica set member as the starting
dbPath for a new member. This approach is included in the tutorial I linked above.