The best you can do with a UIVisualEffectView set to blur is the new blur styles
systemUltraThinMaterial, systemThinMaterial, systemMaterial, systemThickMaterial and systemChromeMaterial (and their light and dark variants.)
The blur amount is the least for the ultra thin material styles, and the most on the thick material styles, but that isn't much control. Sometimes I want a very subtle blur. Even the systemUltraThinMaterial applies a LOT of blur.
Based on another answer to this subject here on SO, I created my own blur view. Rather than creating a view that is placed on top of another view, I created a custom subclass of UIView that blurs it's contents. You put whatever subviews you need inside the view, and it adds a blurred layer on top of the view's contents.
NOTE: As Matt pointed out in the comments, my BlurView class does not provide live blurring. It takes a snapshot of the view's contents and subviews, blurs that, and installs the results as a layer on top of the view's content layer. If you need live blurring of the blurred views this solution is not for you.
Here is a github repo with a working example using my BlurView class:
https://github.com/DuncanMC/BlurView.git
To use it, simply add a the BlurView.swift file to your project.
Then add a UIView to your storyboard and put whatever you want to blur inside it as subviews. (Alternately you could subclass BlurView)
At runtime the BlurView adds a layer on top of the view's content layer that contains a blurred version of the view's contents. The blurLevel
property lets you control the radius of the blur effect applied to the image. A value of less than 1 provides a very subtle amount of blur, and the default value of 10 provides a fairly strong amount of blur.
The blur
boolean turns the blur effect on and off.
The demo app shows a picture, some text, and a view with a background layer and border color.
Here is what the sample view looks like with blurring turned off:

And what the BlurView looks like with a blur radius of 5 (Which I consider to be a fairly strong amount of blurring: )

For comparison, the lightest blur amount for the new-to-iOS 13 "material" setting on a UIVisualEffectView, "systemUltraThinMaterial", looks like this:

(The ultra thin material blur option seems to use a blur radius of around 30)
As Leo points out in his comment, the adaptive forms of UIVisualEffectView applies a lightening or darkening to the blur that follows the user's light mode or dark mode setting. If that's important to you, you might want to use the UIVisualEffectView.
I suspect it wouldn't be that hard to add a lighten or darken step to my blur view and make it honor the user's light mode/dark mode setting. Core Image has lots of different filters and it' pretty easy to combine them.
Edit:
Based on LeoDabus's comment below, I decided to add an exposure compensation filter to the BlurView. Below is a screenshot of the sample app with that option. The top image is my BlurView with a blur radius of 25 and an EV value of -1, and the bottom is a UIVisualEffectView in systemUltraThinMaterialDark style. I think they look quite close - but my BlurView lets you get an infinite variety of different looks based on your needs.

Edit #2:
After Matt mentioned UIImageEffects in his comments, I found a Swift version of that Apple sample code. It's quite easy to use, and a lot faster than the blur Core Image filter. (Note that the Swift version of UIImageEffects left the default "Copyright <authorname>
. All rights reserved." copyright notice, so I would not recommend using it in your projects without getting permission from the author.)
I updated my sample project to have "Use Core Image" UISwitch. When you turn that switch off, it uses UIImageEffects to do the blurring.
The UIImageEffects library has some presets that create looks that approximate light and dark blur modes, but it also includes another of different options. My sample app is currently set up to imitate the light style at the specified blur radius. (It ignores the brightness adjustment setting in UIImageEffects mode.)
The updated version is in a branch called UIImageEffects in the same repo at https://github.com/DuncanMC/BlurView.git