0

I have challenged myself to code an in house app similar to Google's assistant voice tapping app. Everything is finished except the auto tap feature. When I launch the app it spins up a foreground service that draws a 8x8 grid that is always on top. Even when the main activity is minimized. The service uses an imageview that is set drawable. I then wrote an algorithm that draws a grid within it numbered 1 through 64. The user says which number to tap and the service does just that but nothing happens. I Log.wtf the tap coordinates x,y and it works perfectly when my main activity is open. when minimized it doesnt log the event OR its taping the wrong view.

Any suggestions?

Here is the code inside my custom window class that pops the window and sets the tap transperency so that taps pass through to the target app behind it. tPars is different for testing purposes. Both are window parameters.

`

mPars=new WindowManager.LayoutParams(
        WindowManager.LayoutParams.MATCH_PARENT,
        WindowManager.LayoutParams.MATCH_PARENT,
        WindowManager.LayoutParams.TYPE_APPLICATION_OVERLAY,
        WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE,
        PixelFormat.TRANSLUCENT);

lInflater=(LayoutInflater) context.getSystemService(Context.LAYOUT_INFLATER_SERVICE);

gridView = lInflater.inflate(R.layout.window_overlay,null);

iv = gridView.findViewById(R.id.iv);

ivHolder= gridView.findViewById(R.id.ivHolder);
((ImageView)iv).setImageDrawable(new Draw());

mPars.gravity=Gravity.CENTER;
mWManager=(WindowManager) context.getSystemService(WINDOW_SERVICE);

`

Ive even tried using System.exec("Touch X,Y"). Still same issue. Broadcast receivers wont work as the goal is to stay within the service WHILE using other apps. Its an auto tap assistant app like googles.

I would like to utilize this ---> MOTION EVENT

To reiterate, Once Main Activity spins up the service I no longer want or need it as the user will be using other apps. My app allows them to Tap/Scroll with their voice via the foreground service overlay.

Thanks in advance.

I tried MotionEvent, System.exec, and a few 3rd party APIs. It will only tap withing Main Activities view.

EDIT

Here is my tap and recognition code.

else if(sentence.get(0).equalsIgnoreCase("tap")){

                    mWManager.addView(gridView, mPars);
                    grid.draw(new Canvas());

                    tap(548F,946F);
            }

and execution

private void tap(Float x, Float y) {

            gridView.dispatchTouchEvent(MotionEvent.obtain(SystemClock.uptimeMillis(), SystemClock.uptimeMillis()+100, (int)MotionEvent.ACTION_DOWN, x, y, 0));
            gridView.dispatchTouchEvent(MotionEvent.obtain(SystemClock.uptimeMillis(), SystemClock.uptimeMillis()+100, (int)MotionEvent.ACTION_UP, x, y, 0));
        }
Tyler Yarborough
  • 3
  • 2
  • If for internal distribution only, two FYIs: https://source.android.com/docs/devices/automotive/voice/voice_interaction_guide and https://developer.android.com/guide/topics/ui/accessibility specifically https://developer.android.com/guide/topics/ui/accessibility/service – Morrison Chang Nov 28 '22 at 00:34
  • Im not distributing it anywhere. It a challenge and resume item. Is there any way a running service can tap the screen while main activity is minimized? The service utilizes and swaps between 3 views flawlessly it just will not tap. AS kind of defeats the purpose of challenging myself to think outside of the box. – Tyler Yarborough Nov 28 '22 at 00:52
  • To me you are describing [tapjacking](https://developer.android.com/topic/security/risks/tapjacking) where your app may not be malicious but others can be. – Morrison Chang Nov 28 '22 at 01:17
  • According to the litterature its now blocked system level. That explains so much. Thank you friend, if I did submit the APK somewhere they would black list me. Still kinda cool I built something hacky not trying to. – Tyler Yarborough Nov 28 '22 at 01:46

0 Answers0