I think the way to look at this is that it's a misuse of a date components formatter. This isn't a date of any kind. It's a string consisting of a certain number of minutes, seconds, and milliseconds. Unlike date math, that's a calculation you can perform, and then you are free to present the string however you like.
If you want to use a formatter to help you with user locales and so forth, then you are looking for a measurement formatter (for each of the substrings).
Example (using the new Swift 5.5 formatter notation):
let t1 = Measurement<UnitDuration>(value: 2, unit: .minutes)
let t2 = Measurement<UnitDuration>(value: 4, unit: .seconds)
let t3 = Measurement<UnitDuration>(value: 345, unit: .milliseconds)
let s1 = t1.formatted(.measurement(width: .narrow))
let s2 = t2.formatted(.measurement(width: .narrow))
let s3 = t3.formatted(.measurement(width: .narrow))
let result = "\(s1) \(s2) \(s3)" // "2m 4s 345ms"
Addendum: You say in a comment that you're having trouble deriving the number milliseconds. Here's a possible way. Start with seconds and let the Measurement do the conversion. Then format the resulting value in the formatter. Like this:
let t3 = Measurement<UnitDuration>(value: 0.344657, unit: .seconds)
.converted(to: .milliseconds)
// getting the `0.xxx` from `n.xxx` is easy and not shown here
let s3 = t3.formatted(.measurement(
width: .narrow,
numberFormatStyle: .number.precision(.significantDigits(3))))
You might have to play around a little with the number-formatter part of that, but the point is that a measurement formatter lets you dictate the number format and thus get the truncation / rounding behavior you're after.