If the browser or client can play the audio locally then it has to download it - it is possible to do things like obfuscate the URL etc but these are generally easily worked around.
The usual next defence is to encrypt the content and pass the key securely, either with some secure proprietary method or using a DRM system.
For audio, however, because the audio path itself is mostly insecure in devices once the audio is playing, as you have noted, you have to balance the effort/cost vs the extra security this will give you.
The main commercial DRM systems in use at the time of writing are Widevine from Google, FairPlay from Apple and PlayReady from Microsoft. The coverage is roughly:
- Android devices - Widevine
- Chrome browser on a PC or MAC - Widevine
- iOS device - FairPlay
- Safari browser - FairPlay
- Internet Explorer browser, xBox - PlayReady
As you can see, to get wide coverage using these default DRM's you typically need more than one DRM, and often a third party Multi DRM service will be used.
There are also some less secure DRM techniques which don't require a commercial service and typically transmit the key in the clear, but which may meet your needs. These include:
- Clearkey with DASH
- HLS AES Encryption with HLS
You can see some examples here: https://stackoverflow.com/a/46666226/334402