On working on a kiosk style app I know that some Dialogs come to the foreground and can be detected by
ActivityManager activityManager = (ActivityManager)getBaseContext()
.getSystemService(Activity.ACTIVITY_SERVICE);
String className = activityManager.getRunningTasks(1).get(0).topActivity.getClassName();
An example for that is the bluetooth-binding dialog that brings the com.android.settings to the foreground.
A counter-example is the power-button dialog (Turn off, Reboot etc) that does not come to the foreground.
Note that you can close system dialogs (even the power-button dialog) with this broadcast:
Intent closeDialog = new Intent(Intent.ACTION_CLOSE_SYSTEM_DIALOGS);
sendBroadcast(closeDialog);
But on most (all newer?) devices this Broadcast will even close the software keyboard, so it is not advisable to have a service running that frequently sends it as the user will then be unable to enter anything into a text field.
Note that such behaviour will definetly gratify your app a status as beeing malware, keeping it from beeing published on google play.