It could be argued that a unit-test library is just a library, but I don't see it like this.
First, the purpose of a unit-test library is to be used in code that is not production code. Which means, that certain quality criteria relevant for production code might not be met. For example, if there is a bug in a unit-test library it is annoying, but normally does not harm the production code. Or, performance may not be quite as relevant, thread safety and so on. I don't want to say that the popular unit-testing frameworks are of a bad quality. But, the developers of these libraries have all the right in the world to take design decisions based on the assumption that their code will not be part of production code.
Secondly, using a library should be done in accordance to the philosophy of the respective library. For example, if you use a certain gui library, this brings implications on the way event handling is done in your application. And, unit-testing frameworks come under the assumption that the framework is in control of the executable (from the test runner). Therefore, all functions from that library may depend on the test runner being set up and running. If some function from the library does not have this dependency, that is an implementation detail which may change with a new version of the library.
Finally, code should communicate intent. That includes includes (pun intended). It was not the intent of the developer to write unit-testing code, but that would be what the inclusion of a unit-testing library would communicate.