Take a look here: how much abstraction is too much?
Basically you need to do some sort of cost-benefit analysis. If you can easily understand the abstraction and it makes tangible benefits to your game, then go for it. If it starts to make things unnecessarily complicated and it makes it hard to understand what it being abstracted, then its probably not worth it. There's a point of diminishing returns, but finding it and justifying whether another layer of indirection is a good idea or not is a difficult art.
Is this good practice at all to
abstract all libraries first before
getting into the actual development?
If you are the only developer then its going to revolve largely around your personal preference. If you are familiar with the library and can tell what's going on easily, then you're fine. If others are involved, then it depends on how easy your abstraction is to use and if learning it better than using the actual library without any addition layers.