Split your library in two and remove the reference for log4net.dll from the main library, and reference the second library only if you'll need stuff tied to part that need log4net.dll.
Good way how to handle such cases is dependency injection - take a look at Enterprise Library (unity container) - though that will give you one extra dll :)
Using Unity Container:
You'll have Library1 with ILog4NetUsingInterface
In Library2 you'll have class Log4NetUsingClass : ILog4NetUsingInterface
In Library2 you'll have bootstrapper that will register Log4NetUsingClass as implementation of ILog4NetUsingInterface
public static class Bootstrapper {
public static void Init() {
ServiceLocator.IoC.RegisterSingleton<ILog4NetUsingInterface, Log4NetUsingClass>();
}
}
This Init method you'll call only if you need to use the Log4NetUsingClass
In every other library you can just call
ServiceLocator.IoC.Retrieve<ILog4NetUsingInterface>()
(If you won't call bootstrapper it will give you runtime error.)