We had the same problem and we fixed it. Twice.
Incremental build (same build machine):
before: ~10m
after: ~35s
HOW?
Let's start with our experience first. We had a massive Swift/Obj-C project and that was the main concern: build times were slow and you had to create a new project to implement a new feature (literally). Bonus points for never-working syntax highlighting.
Theory
To truly fix this you have to truly understand how build system works.
For example, let's try this code snippet:
import FacebookSDK
import RxSwift
import PinLayout
and imagine you use all of these imports in your file. And also this file depends on another file, which depends on another libraries, which in turn uses another libraries etc.
So to compile your file Xcode has to compile every library you mentioned and every file it depends on, so if you change one of the "core" files Xcode has to rebuild literally whole project.

Xcode build is multi-threaded, but it consists of many single-threaded trees.
So on the first step of every incremental build Xcode is deciding which files have to be re-compiled and builds an AST tree. If you change a file which is acting as "dependable" on other files, so every other file which acts as "dependent" has to be recompiled.

So the first advice is to lower coupling. Your project parts have to be independent of each other.
Obj-C/Swift bridge
Problem with those trees if you're using a Obj-C/Swift bridge, Xcode has to go through more phases than usual:
Perfect world:
- Builds Obj-C code
- Build Swift code

Obj-C/Swift bridge:
- [REPEATABLE STEP] Build Swift code, which is needed to compile Obj-C code
- [REPEATABLE STEP] Build Obj-C code, which is needed to compile Swift code
- Repeat 1 & 2 until you have only non-dependable Swift & Obj-C code left
- Build Obj-C code
- Build Swift code

So if you change something from step 1 or 2, you're basically in a trouble.
The best solution is to minimize Obj-C/Swift Bridge (and remove it from your project).
If you don't have an Obj-C/Swift Bridge, that's awesome and you're good to go to the next step:
Swift Package Manager
Time to move to SwiftPM (or at least configure your Cocoapods better).
Thing is, most frameworks with default Cocoapods configuration drag along with themselves a lot of stuff you don't need.
To test this create an empty project with only one dependency like PinLayout, for example and try to write this code with Cocoapods (default configuration) and SwiftPM.
import PinLayout
final class TestViewController: UIViewController {
}
Spoiler: Cocoapods will compile this code, because Cocoapods will import EVERY IMPORT of PinLayout (including UIKit) and SwiftPM will not because SwiftPM imports frameworks atomically.
Dirty hack
Do you remember Xcode build is multi-threaded?
Well, you can abuse it, if you are able to split your project to many independent pieces and import all of them as independent frameworks to your project. It does lower the coupling and that was actually the first solution we used, but it wasn't in fact very effective, because we could only reduce incremental build time to ~4-5m which is NOTHING compared to the first method.