Is Monkey really meaningful?

Question

As part of a new Android application, we are setting up a full Continuous Integration (with Jenkins, but it doesn't matter).

We have automatized JUnits and UI tests as of now, and we were thinking about adding Monkey testing after each build, just as an additional quality step.

However our feedback on Monkey is quite negative so far. We have created a freshly new, empty Android application (the basic Hello World app when you create an empty project in Android Studio), and even then, a Monkey test regularly crashes (running on an emulator), for what appears to be unrelated to our application.

The command I tried:

adb shell monkey -v -v -s -2972043913753481246 -p my.app.package 50000

But it crashes along the way (around step 6000).

I thought this may be due to the emulator being over-solicited, so I added some throttling to let the emulator breathe:

adb shell monkey -v -v -s -2972043913753481246 --throttle 150 -p my.app.package 50000

But still, it crashes (further, around step 8000).

I thought that it maybe needed even more breathing, so I put a 1 second throttling just to make sure (even humans can act faster than that)

adb shell monkey -v -v -s -2972043913753481246 --throttle 1000 -p my.app.package 50000

And still, it crashes (around step 48 000).

Adding the --ignore-native-crashes did not change anything.

So my whole point is: does Monkey really make sense anyway? It doesn't appear to be relevant (if an empty Hello World app provided by Android Studio itself is the root cause, then I don't want to see what would happen with a real-world app).

Did somebody find a way to make Monkey testing stable and relevant? What configuration (number of events, throttling, additional flags) do you use?

One of the crashes I get with 150 ms throttling:

// CRASH: com.android.launcher (pid 1838) // Short Msg: java.lang.IllegalArgumentException // Long Msg: java.lang.IllegalArgumentException: width and height must be > 0 // Build Label: generic_x86/sdk_google_phone_x86/generic_x86:5.1/LKY45/1737576:eng/test-keys // Build Changelist: 1737576 // Build Time: 1423932217000 // java.lang.IllegalArgumentException: width and height must be > 0 // at android.graphics.Bitmap.createBitmap(Bitmap.java:810) // at android.graphics.Bitmap.createBitmap(Bitmap.java:789) // at android.graphics.Bitmap.createBitmap(Bitmap.java:756) // at com.android.launcher2.Cling.dispatchDraw(Cling.java:201) // at android.view.View.updateDisplayListIfDirty(View.java:14162) // at android.view.View.getDisplayList(View.java:14189) // at android.view.ViewGroup.recreateChildDisplayList(ViewGroup.java:3389) // at android.view.ViewGroup.dispatchGetDisplayList(ViewGroup.java:3368) // at android.view.View.updateDisplayListIfDirty(View.java:14127) // at android.view.View.buildLayer(View.java:13979) // at com.android.launcher2.AppsCustomizeTabHost.enableAndBuildHardwareLayer(AppsCustomizeTabHost.java:359) // at com.android.launcher2.AppsCustomizeTabHost.onLauncherTransitionStart(AppsCustomizeTabHost.java:403) // at com.android.launcher2.Launcher.dispatchOnLauncherTransitionStart(Launcher.java:2538) // at com.android.launcher2.Launcher.hideAppsCustomizeHelper(Launcher.java:2852) // at com.android.launcher2.Launcher.showWorkspace(Launcher.java:2900) // at com.android.launcher2.Launcher.showWorkspace(Launcher.java:2893) // at com.android.launcher2.Launcher.startSearch(Launcher.java:1642) // at com.android.launcher2.Launcher.onSearchRequested(Launcher.java:1766) // at com.android.launcher2.Launcher.onKeyDown(Launcher.java:891) // at android.view.KeyEvent.dispatch(KeyEvent.java:2619) // at android.app.Activity.dispatchKeyEvent(Activity.java:2707) // at com.android.launcher2.Launcher.dispatchKeyEvent(Launcher.java:1973) // at com.android.internal.policy.impl.PhoneWindow$DecorView.dispatchKeyEvent(PhoneWindow.java:2276) // at android.view.ViewRootImpl$ViewPostImeInputStage.processKeyEvent(ViewRootImpl.java:4020) // at android.view.ViewRootImpl$ViewPostImeInputStage.onProcess(ViewRootImpl.java:3982) // at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:3544) // at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:3597) // at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:3563) // at android.view.ViewRootImpl$AsyncInputStage.forward(ViewRootImpl.java:3680) // at android.view.ViewRootImpl$InputStage.apply(ViewRootImpl.java:3571) // at android.view.ViewRootImpl$AsyncInputStage.apply(ViewRootImpl.java:3737) // at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:3544) // at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:3597) // at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:3563) // at android.view.ViewRootImpl$InputStage.apply(ViewRootImpl.java:3571) // at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:3544) // at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:3597) // at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:3563) // at android.view.ViewRootImpl$AsyncInputStage.forward(ViewRootImpl.java:3713) // at android.view.ViewRootImpl$ImeInputStage.onFinishedInputEvent(ViewRootImpl.java:3874) // at android.view.inputmethod.InputMethodManager$PendingEvent.run(InputMethodManager.java:2208) // at android.view.inputmethod.InputMethodManager.invokeFinishedInputEventCallback(InputMethodManager.java:1849) // at android.view.inputmethod.InputMethodManager.finishedInputEvent(InputMethodManager.java:1840) // at android.view.inputmethod.InputMethodManager$ImeInputEventSender.onInputEventFinished(InputMethodManager.java:2185) // at android.view.InputEventSender.dispatchInputEventFinished(InputEventSender.java:141) // at android.os.MessageQueue.nativePollOnce(Native Method) // at android.os.MessageQueue.next(MessageQueue.java:143) // at android.os.Looper.loop(Looper.java:122) // at android.app.ActivityThread.main(ActivityThread.java:5257) // at java.lang.reflect.Method.invoke(Native Method) // at java.lang.reflect.Method.invoke(Method.java:372) // at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903) // at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698) //

The crash I get with 1000ms throttling is way to huge (tens of thousands of lines) to post on SO, I fear.

Monkey is relevant because it causes the worse case unpredictable nature of a user and ensure it doesn't crash. Also without what the crash is.. its hard to say. — JoxTraex, Aug 27 '15 at 08:52
I would rather get I human to do my testing and not a monkey, but that's just me ;) — Smashing, Aug 27 '15 at 08:58
No reason why you can't have monkey and a human test it, the random factor is what makes the difference @Smashing :) — JoxTraex, Aug 27 '15 at 09:00
@Smashing I would rather get both. Humans are biased, if they see a button, they won't naturally click somewhere else. I think it's a nice complementary solution... When it's reliable and stable :/ — Sir4ur0n, Aug 27 '15 at 09:02
That is weird though, why are touch events being dispatched to the Launcher2? Generally monkey should only go to your app. I would figure the only way this could happen is your app is going to the background because home was it or something of the sort, have you looked at what the device is doing just before it transitions into this state? — JoxTraex, Aug 27 '15 at 09:25
@JoxTraex That's what surprised me. Even with my app package specified via -p option, it goes to the launcher and the home pages. Isn't it supposed to happen? — Sir4ur0n, Aug 27 '15 at 09:28
Hmm, just tried it myself it it seems touch events do go to the launcher — JoxTraex, Aug 27 '15 at 09:35
If you think humans are biased then you haven't met our QA team yet.. — Smashing, Aug 27 '15 at 09:52
Actually I think I found the root problem is not Monkey per se, but the Android emulator... Because I just tested with 2 physical devices (a Nexus 5 and a Samsung Galaxy Ace) and I don't encounter any issue with them. Additionally, Monkey never leaves the app on these devices, but Monkey on an emulator also goes to the Launcher part (i.e. not in the app)... Should I create a new question and close this one? — Sir4ur0n, Aug 28 '15 at 13:54

score 0 · Answer 1 · edited Sep 22 '16 at 06:27

0

The interest thing is Monkey is unpredictable, you never know what happens next. Monkey performs random events, if lucky it can trigger all events without any order, then popups some hide，unexpected exceptions or effects in application.

edited Sep 22 '16 at 06:27

einverne

6,454
6
45
91

answered Aug 27 '15 at 09:05

Lemon

487
4
12

I'm not sure this answers to my question. I am asking if Monkey is meaningful because it isn't stable at all, and brings many crashes which don't seem related at all to our application... – Sir4ur0n Aug 27 '15 at 09:15
Monkey can be limited in your application, so you can filter what cash happens with yours only.Sometimes several continuing events can also cause crash – Lemon Aug 27 '15 at 09:36
As you know,Monkey is limited,I just use it to do some preliminary tests – Lemon Aug 27 '15 at 09:39

score 0 · Answer 2 · edited May 23 '17 at 10:27

I believe the answer is YES to the question "is monkey test meaningful". As you have mentioned that the issue may related to the bug of emulator. And I suppose that your monkey test command is limited in your application and all settings are ok.

And jump out of this bug, you can see that monkey test is a kind of stress testing. It can identify some out-of-the-box errors. And there are some reasons, I take from wiki:

The intrinsic randomness of monkey testing also makes it a good way to find major bugs that can break the entire system.

Smart monkeys, if properly set up with an accurate state model, can be really good at finding various kinds of bugs.

So monkey test as part of software testing is necessary. And if you want your monkey become "smart". You can use the monkeyrunner provided by Google to specify some event and manually test part of your application. Monkeyrunner may be a little bit stable and relevant. Here is some tips and tricks I used in Android monkey test.

Is Monkey really meaningful?

2 Answers2