The refresh token needs to be managed and handled in a secure way. Preferable, the refresh token should not end up in the browser. Also the refresh token is never send to the APIs.
So the refresh token is something that is only sent between the client and the authorization server (Not the browser or API's)
So, if you do everything properly, the refresh token never leaves the backend (not exposed to the browser), then it is pretty secure (unless your backend is hacked, but then you are game over anyway).
You can further strengthen it by securing the communication channel between the client backend application and the authorization server using certificates and other networking restrictions. So, if you follow the best practices, as outlined in OAuth 2.1, it is pretty safe and solid.
So, the conclusion is that as the refresh token is never used outside the client who originally received it, its pretty safe and as long the backend is not hacked, then its secure.
If you are building a SPA client (React...) then you should consider using the BFF pattern. As handling of tokens in the browser is never a good idea. Also see this video