Edit
Sorry, I didn't notice that you don't want to support different versions of the library. In this case there is no way — when FP processes the bytecode of an SWF and when it finds an unknown reference, it throws VerifyError. Of course, you can use getDefinitionByName() and dynamic access, but it's very slow.
To minimize the amount of separately supported code, you can use an SWC with main functionality and SWFs with version-dependent functionality, so at the stage of initialization the main class of your lib will check FP version and load the appropriate SWF.
This is called dynamic linking, and there is a way:
Create an interface of all player version dependent functionality (public interface MyInterface...). This interface should not contain any references to version dependent APIs. Compile this interface (it may be just single .as file) into SWC (let it be lib-intf.swc).
Create two independent implementations of this interface; the first will use the new APIs, the second won't (it may be just a placeholder, but also may be an alternative implementation). Later we'll need to compile these implementations into SWFs, so we need main classes that extend Sprite. The simpliest way to do the trick is to make these main classes implement our interface (i.e. public class MyImplementationA extends Sprite implements MyInterface..., and the same for MyImplementationB). They will be just empty Sprites, but they will contain interface methods.
Compile these two implementations independently into separate SWFs (lib-a.swf and lib-b.swf). When compiling, include lib-intf.swc as an external library (-external-library-path compiler parameter or "external" linking type in IDEs).
Now, when compiling your root application, include lib-intf.swc as usual library (-library-path compiler parameter or "merged into code" linking type in IDEs). Don't include version-dependent classes at all. Thus, in the root application you will have just references to the interface, which is version-independent. When your application starts, check FP version, and depending on it load appropriate SWF using Loader class. You'll have to load it into the main application domain, not a child of it (which is default option; more details).
When the SWF is loaded, cast it to the interface: var versionDependentImpl:MyInterface = loader.content as MyInterface. Remember that main classes of our SWFs implement MyInterface, so this cast would work.
That's it — now you can use your implementation: versionDependentImpl.someMethod(). Of course, someMethod should be defined in MyInterface.
So, the trick here is to dynamically load implementation from an SWF file. Though the root application doesn't know anything about classes inside this SWF, we can use methods of it's main class because we made it implement an interface that we have compiled into the root application.
This approach is scalable: for example, you can define main interface which have methods that return other interfaces. You can even include concrete classes that are shared between implementations into lib-intf.swc, as long as they don't use version-dependent APIs.
