-3

Recently in QCon, Gerard Meszaros said that we should run automation tests only on simulators to improve efficiency. We are also considering about the mobile automation test. But I'm not sure if there will be some issues that can only found in a real device? Or some components like camera, gravity sensors could not be tested in a simulator/emulator? Thanks!

Edit:

You can find Gerard's slides here: http://www.slideshare.net/pinot_noir/ensuring-quality-in-cloud-and-mobile-applications.

I read through the slides again and I found that I misunderstood his point. What he really means is that we should try to avoid debugging on the hardware (page 41). And he also mentioned we could trap outgoing requests to the sensors and other services, then inject the response from them. Maybe we can pick some cases that couldn't be fully tested in simulator and then test it with real devices.

Thank you for all your comments! :)

010Bytes
  • 105
  • 1
  • 8
  • This is way too broad, don't you have a more specific type of application in mind? – KillianDS Dec 09 '12 at 15:29
  • 1
    A statement such as testing should _only_ be run on simulators is a statement from someone who should be summarily ignored. Simulation definitely has its place in testing however no one with any degree of competence would state that simulation is valid for all testing. Also, no one with competence would suggest that simulation is even more efficient in all cases, since depending on the sim it can easily be slower. For example, Android simulation is generally slower than the device, while iOS simulation provides a close but not exact environment. – mah Dec 09 '12 at 15:36
  • @mah android simulation is slow..because it is not simulation it is `emulation`:) – Mohsin Naeem Dec 09 '12 at 15:45
  • @mah, efficiency is not always about test execution speed. It could very well be about cost for example. In many situations (and I'm talking even beyond smartphones here), a sim environment cost is significantly lower then a real hardware environment. Also, simulation tends to be more easy to integrate into automation environments. In the end these are often critical factors for a QA environment, test duration is important, but not top of the stack. – KillianDS Dec 09 '12 at 19:04
  • 1
    @KillianDS yes, of course, yet another reason why blanket statements such as "should only be tested on ..." are laughable at best. – mah Dec 09 '12 at 20:33

1 Answers1

1

Recently in QCon, Gerard Meszaros said that we should run automation tests only on simulators to improve efficiency.

This was odd advice, if that is really what Mr. Meszaros said. Running tests on the emulator is fine, but "only" is an excessive recommendation. There is no harm in running automated tests on devices, and you can learn a lot from doing so.

But I'm not sure if there will be some issues that can only found in a real device?

Of course.

  • Many devices have multi-core CPUs, whereas the emulator only emulates a single core at this time
  • Device storage tends to run a lot slower than does storage on the emulator
  • Device manufacturers tinker with Android in ways that will not appear on an emulator running stock Android
  • The emulator only loosely emulates hardware related to power, Internet (e.g., no mobile data, no WiFi), GPS, sensors, camera, etc.
  • The emulator does not support some device capabilities, like the new V2 version of Maps, the Play Store, multiple accounts on Android 4.2, etc.

And so on.

Or some components like camera, gravity sensors could not be tested in a simulator/emulator?

Those ones are difficult to test in an automated fashion, period.

CommonsWare
  • 986,068
  • 189
  • 2,389
  • 2,491
  • According to http://stackoverflow.com/a/13734937/14731 the emulator can be made to run Android Maps v2. – Gili Dec 10 '12 at 01:55
  • @Gili: Only by means of software piracy. – CommonsWare Dec 10 '12 at 11:07
  • I think it's a gray area. It's pretty clear that Google plans on providing this functionality in the near future. It's also pretty clear that Developers are using this in "good faith" as a workaround until Google comes out with an official release. I haven't heard of anyone using this in "bad faith", have you? – Gili Dec 10 '12 at 22:19
  • @Gili: "I think it's a gray area" -- the law would tend to disagree with you. "I haven't heard of anyone using this in "bad faith", have you?" -- most software piracy is conducted in "bad faith" IMHO. However, my ire comes from software developers -- who generally complain loudly when *their* apps are pirated -- recommending piracy as a solution for perceived problems. At some point, hopefully Google will release Maps V2 for the emulator. Until then, use hardware, or avoid Maps V2. – CommonsWare Dec 10 '12 at 22:33
  • How is that a fair comparison? Developers complain when their apps are distributed in a manner that harms them, mostly financially. How is Google being harmed here (financially or otherwise)? Google knows exactly who is distributing these files and yet they haven't even sent a cease and decease letter to them. Why is that? I don't think Google would complain against this behavior even if they could do it for free without any negative publicity. I don't see the value of being outraged on Google's behalf when they aren't outraged themselves. – Gili Dec 11 '12 at 03:47