It all depends on specific use cases.
As other people pointed, in many cases you don't need to hold A reference in your listener and instead rely on having it being passed in the event itself. But this is not always enough - for example, you might want to deregister yourself depending on some other event and you need original reference to call removeListener.
But this is more generic question - if you still hold reference to A somewhere else, it won't get GCed regardless of your listener design. On the other hand, if you have forgotten references to A everywhere in your program and don't hold reference to B as well, they will be both garbage collected anyway - circular references are not preventing garbage collection in java (How does Java Garbage Collection work with Circular References?)
I had bit similar situation in the past, but I cared about B being GCed rather than A. When all references to B from outside are lost, there is nobody to unregister it, but it is still getting notified from A on every event. It is quite common to end with such danging listeners in UI frameworks in java if you don't clean up after yourself perfectly (which people often don't do, as they assume that getting rid of the UI component graphically and forgetting all references to it should be enough - while for example, some listener to global keyboard handler or something still holds everything in strongly reachable set).
While for UI frameworks you don't get a chance to change core code, with your own code you can try to make listener lists having WeakReference instead of strong references and do some cleanup on every notification if needed. It has a side effect of you having to keep your listeners references from other places of code, but it is a good practice anyway (as you should be deregistering them manually in other case) - if you don't do that, with weak reference you would suddenly stop getting called back at random times (after few gc cycles have run).
In any case, you first need to understand (and tell us), how do you imagine lifecycle of A and B and why B will be strongly referenced after A could already disappear. You can always make B->A reference weak as well, but please first be sure to fully understand your expectations about what gets forgotten at what point.