0

I want to detect gestures that's done by user without touching screen, i.e., the user just swipes his hand over(for example, 3 cm from the screen) the screen of an Android device from the speaker to the microphone. I tried the Sensor type TYPE_PROXIMITY, via which it seems the device knows user just put his hands near the device and then off it. But I want to know the direction of the gesture, that is, whether the user swipes his palm from the speaker to the microphone, or the opposite. Should I try some other Sensor type? Or need I combine multiple types of Sensor to achieve it?

Briefly, I want to know: 1. How to detect user swiping over the screen? 2. How to detect the direction of the swipe?

PS: it's not gesture detecting of screen touching, which I can handle by implementing GestureDetector.

Kindest regards, Nessus.

user1105115
  • 483
  • 2
  • 5
  • 18

2 Answers2

1

Form a user standpoint hovering 1 cm over the screens highly inconvenient compared to placing a finger over the screen. Swipes seen by a front facing camera with a small aperture will be contaminated with the motion blur for a reasonable speed of swipe.

A few years back I solved this problem by considering how motion blur of swipe would eventually occlude the image on the background. In particular, if background has some gradient, it will be erased by the motion blur of the moving hand. Thus if you histogram vertical gradient, you will see a hole moving through your histogram when you swipe and the direction of motion of this hole is your swipe direction.

Two more pointers: if there is not much gradient on the background one can simply reduce image resolution and work with the gradient image of the hand that is less affected by the motion blur at lower resolutions. When camera itself moves the edge histogram will translate globally as opposed to a hole moving through a histogram. Finally, extreme phone rotations can be detected by embeded gyroscope to avoid false positives.

Vlad
  • 4,425
  • 1
  • 30
  • 39
0

You can't detect the motion vector with the proximity sensor. But there usually is a much smarter sensor that allows for much more precision: the front camera. It is more complex to read gestures with a camera, but you can definitely do that with OpenCV, for example.

Rob
  • 11,492
  • 14
  • 59
  • 94
  • thank you. Is it possible to combine proximity sensor with others? Samsung shows possibility in its ads; someone in the kitchen just swipes over the phone on the table since the hands is dirty. If it can be most possibly only achieved by camera, can you share more info about it, such as snippet or link? – user1105115 Nov 06 '13 at 09:14
  • As far as I know, there are newer display types available that can detect gestures in a reasonable height without touching the surface. I can already see this behaviour on my HTC One S, it detects gestures within a range of 1cm above the screen. The OpenCV project offers an impressive set of features, but it's not the easiest topic though definitely worth diving into. Here's a good starting point: http://docs.opencv.org/doc/tutorials/introduction/android_binary_package/dev_with_OCV_on_Android.html – Rob Nov 06 '13 at 09:37
  • 1
    If OpenCV works only within 1cm above the screen, it may not be a user friendly option for interaction. **A range of at most 10cm** is 90% acceptable, I think. – user1105115 Nov 07 '13 at 03:31
  • You got me wrong, I think: OpenCV is a framework for image processing, motion detection and tracking and some other handy features that you can use to work with a live video stream (from a camera, e.g.). It therefore is only limited by the quality of the camera, but that's surely way beyond the 10 cm you speak of. The OTHER way of detecting gestures is the screen itself, which is in my case limited to 1 cm (and obviously more with the new Samsung display). In that case you don't need OpenCV. – Rob Nov 07 '13 at 08:15
  • 1
    Got it, buddy. I will implement screen touching at this phase and try OpenCV afterwards. Thank you for showing the way to the future:) – user1105115 Nov 08 '13 at 01:28
  • BTW, buddy, have you seen this question? http://stackoverflow.com/questions/16649830/proximity-sensor-on-galaxy-s4-air-gestures?rq=1 . That's exactly what I want, air gesture with direction. – user1105115 Nov 08 '13 at 01:41
  • BTW of BTW, I dont know whether it's already available on Samsung devices, but I am about to distinguish brands for this feature if other brands lag behind in providing this fancy APIs – user1105115 Nov 08 '13 at 01:54
  • I don't know for sure, but as far as I can see from the Android API, there is no specification for motion vector detection in the proximity sensor event values, so I do believe that this is a Samsung-specific (and probably camera-based?) feature that you won't find on many devices- – Rob Nov 08 '13 at 07:27