I think it's first of all a question of style and personal preference. It also depends on what your application should do and how.
If your JPanels all are very similar in a certain way it would probably make sense to extend from JPanel. For example if every panel had 10 Buttons where Button 1 does always action xyz() for its corresponding model-object (especially when this object is the same for all buttons of 1 JPanel) and Button 2 does abc() ...
If the JPanels are not strongly correlated in such a way I would place my code rather in the JFrame or in a third Object which sets the whole GUI up from outside.
It's similar with the actionlisteners. If you have very few actions I would probably go for just one controller-object for ease of use (not many files). If you have a lot of different actions I would group similar actions into one controller-object per group of actions.
That said it probably is best to start with a simple approach (YAGNI) where you do everything from the JFrame / third object and have one actionlistener and then refactor when you feel that splitting things up gives you a cleaner or more flexible design.