Simply put, you do not launch any pop-ups from back-end. Your backing beans have no understanding of the DOM or the browser. What you can do, however, is saving the state in a backing bean and render a given HTML or JavaScript block based on that state. For this you can use the rendered="#{mybean.myboolean}"
attribute in conjunction with a update="elementID"
on your button. When clicked, the button will re-render an element with the given ID on the server and inject it into the DOM. If it contains HTML or JS code for a pop-up, then it shows up.
See here for a nice example: Ajax update and submission using h:commandButton
Or you could try the JSF messages component: http://www.mkyong.com/jsf2/jsf-2-message-and-messages-example/
EDIT: since you seem to be using PrimeFaces, you might also want to look into Dialogs and Validation.