As you probably already know, code that does not change does not need to be recompiled. This is true for executables and libraries alike.
A library is supposed to provide you addictional functionality in a neat, pre-packaged form. The difference between additional code you add to your project and a library is that the code included in a library is supposed to be in a stable state, so that once built the user will be able to use its features without any maintenance hassle; the APIs will be available and they will always work. You will be able to discard any implementation files, and just work with the header files - which provide you with the API within your code - and the library files, which contain the compiled implementation.
You are pretty much pre-compiling part of your program; a part that will also be able to be used in other projects, again without recompiling.
Note that C++ itself is an example of this: an implementation of the C++ standard library (for example libc++) is already included with your compiler, so that you are able to use the standard C++ headers without the need to recompile C++ whole every time you try a "Hello World!" program.
You can even extract libraries out of parts of your project that you feel are already completed and stable: this can allow you to reduce the time required to compile your project even though it becomes bigger. These reasons are part of why modularity is so strongly encouraged when programming.
TL; DR: Recompiling a library only once is not only acceptable, is most probably what you want to do.