4

We were exploring various test suites for mobile automated testing and ran into this company called Perfecto Mobile. One of the features that blew me away was they are able to (without jailbreaking) effectively perform a "Remote desktop" on a physical iPad.

So, the iPad's screen is mirrored within a web application, it can register touch / swipe events on the web app and perform them on the device. The only relevant technical detail I have is that all this is being performed using commands sent over the USB cable.

I'm really curious as to how this is implemented and details on relevant Private APIs if any.

Thanks,
Teja

Tejaswi Yerukalapudi
  • 8,987
  • 12
  • 60
  • 101
  • Hi Tejaswi, I am looking for similar kind of solution. Screen-sharing of my iOS app views to a desktop/web based application. Answer in your query helps a lot for analyzing further, but still your help is needed importantly! Please check my query here->http://stackoverflow.com/questions/20054509/ios-convert-ios-app-screen-as-network-packets how can i proceed further? He suggested to have VNC server and use socket connection for transferring data. Where will be this VNC server, On desktop side? Do you have any idea how can i transfer screenshot images to VNC server and then pass to noVNC? – Getsy Dec 10 '13 at 07:06
  • Did the device and web app need to be on the same network? – Adam Johns Jul 31 '14 at 14:00

2 Answers2

5

I'm not familiar with PerfectoMobile, but I can give you a few pointers on how this can be accomplished:

  • For the mirroring, one way would be to look at using AirPlay, the APIs are pretty well documented, but not to do what we're talking about which would require some serious reverse engineering, but it's definitely possible, these guys have done it. A different approach would be to run a background app that would periodically take snapshots of the main screen, and send them over a socket connection to a client. You could do this as a VNC server, and to incorporate the remote view in a web app, you could use noVNC. As far using a USB connection, in the case of the background app talking to a client over TCP, you could to a port forward.

  • To actually perform on the device the touch events sent from your remote viewer, most people have been using the GSEvent group of functions from the GraphicsServices private framework without needing to jailbreak the device. Again, a background app would receive over a socket an instruction such as "Tap there", instantiate the GSEvent, and inject it so it gets processed in the run loop of the most front app.

These few possibilities, at least, have been implemented successfully in different iOS apps up to iOS 6.1 (iOS7 is a different animal). You won't find any such app in the App Store, since Apple clearly prohibits the use of private frameworks in 3rd party apps, instead people deploy them in-house using Enterprise and ad-hoc provisioning profile. On Android however, there's VMLite available in the Play Store.

Frank
  • 110
  • 8
  • Hi Frank, I am looking for similar kind of solution. Your answer helps a lot for analyzing further, but still your help is needed importantly! Please check my query here, please help -> http://stackoverflow.com/questions/20054509/ios-convert-ios-app-screen-as-network-packets http://stackoverflow.com/questions/20445732/screen-sharing-open-source-for-receiving-screenshot-images http://stackoverflow.com/questions/20444284/ios-streaming-network-packets-pcap-usage – Getsy Dec 09 '13 at 19:41
  • @Frank I am facing the issue for iOS 7 using GSEvent. But I read some blogs which are saying to refer to IOHID. But still I don't find solution to send event using IOHID. So any one please help me. – Nishith Shah May 21 '15 at 07:54
0

If you looking to share screen from ios / android, check out skreen.me. They have sample apps you can try out, also they provide libs for mobile app integration.

asm
  • 1