Throttle Live-Migration network traffic

Ops and Admin
,
#1

Hi there,

I plan to use the MongoDB Atlas Live-Migration to move a local ReplSet to Atlas. There should be no downtime. So the Live-Migration comes in handy. But I have only a small line of 100MB/s available which is used by ~30% constantly. I fear when I live-migrate that I will have a major impact on current processes which need bandwidth. The DB is > 500GB so even when would use 100% of the line this is > 12h
So in case I would impact my current processes for > 12h I would better go for mongorestore, which uses compressed data but this would still be quite a big downtime. This currently not an option. I would like to have the smallest downtime window possible, I do not care how long the Live-Migration takes so even 24h would be ok.

TL;DR:
How can I limit the traffic of the MongoDB Live-Migration (monogmirror) to max 50% of my available network bandwidth capacity? I did some investigation with tc and trickle but both seem not to be the correct tools, or I misunderstood how to use them.

Regards,
Michael

#2

Hi @Michael

On the mongomirror side adjusting numParallelCollections to a lower number may correspondingly lower bandwidth consumption, but this would be non deterministic.

tc won’t allow you to rate limit but you could arrange the qdisc and prio to lower the priority of atlas destined packets.

trickle looked like the userspace tool to use, seems to work okay with scp for example:
trickle -v -u 50 -s scp Ontario.zip destination:

#3

Hi @Michael,

If you need to throttle bandwidth or guarantee quality of service for important use cases, I would look at VLANs and QoS settings available via your network or router config. Reducing the number of parallel collection migrations may help, but without explicit traffic shaping you are relying on the cooperative nature of TCP. This will may not be ideal if the 30% of active usage includes latency-sensitive applications or interactive sessions.

Something like tc or trickle should also work, but I personally find VLAN & QoS more straightforward to configure if there is a management interface. I noticed that trickle does not support statically linked executables, which may preclude usage with mongomirror.

Make sure you are using the latest version of mongomirror. It supports compression by default:

As of version 0.9.0, mongomirror uses wire compression if it is enabled with either the source or the target. Use the --compressors option to specify which compression libraries to allow.

Can you share more information on what you tried and what didn’t work?

You may also be able to get more detailed advice on traffic shaping from a community like Server Fault. Applications generally try to use as much resource as the O/S or network allows, and more nuanced management happens at a layer which has more visibility on overall activity and priorities.

Regards,
Stennie

#4

Hello,

I think it was more confusing than productive to mention mongomirror

The use case is to utilize the MongoDB Atlas Live-Migation to migrate a local DB to MongoDB Atlas. MongoDB Atlas will use mongomirror. The local DB is uncompressed appr. 500 GB and the line is 100MB/s from which 30% -50% need to be available for daily business. In this scenario I like to make sure that the Live-migation only take max 50% of the line. When the live-migration takes much longer that is no issue, I will take care that I have an oplog big enough to cover a potential resume of the Live-Migation.

Regards,
Michael

#5

Hi Michael,

I think the mongomirror context is more helpful than confusing, and your general use case of traffic shaping seems clear.

However, the elaboration I was looking for was around tc and trickle not working to limit bandwidth as you expected. If you can share the command-line incantations and outcomes that didn’t work, perhaps someone will have suggestions on how to adjust those.

I don’t find tc and iptables incantations particularly intuitive to decipher, which is why I usually look for alternatives like VLANs or QoS configured through networking devices rather than fussing with Linux command lines.

You might find some useful tc info in the following references:

I assumed you were running Linux based on the mention of tc, but what specific distro and version are you using?

Regards,
Stennie