How to get code coverage reports from google Firebase for Android Espresso tests

Question

Based on this documentation - https://developer.android.com/studio/test/command-line.html#AMOptionsSyntax it is possible to get code coverage results back from Firebase lab. Some folks in #test-lab at firebase-community.slack.com are able to get it working but after few attempts I am still hitting a wall.

I am able to get a combined code coverage report of jacaco and emma by following this guide so there is nothing wrong with my local setup but problematic when trying to give arguments to gcloud cmd line to ask for coverage numbers and it talks about emma coverage.

Essentially, when I run this command locally

gcloud beta test android run \
  --type instrumentation \
  --app app/build/outputs/apk/*-debug-unaligned.apk \
  --test app/build/outputs/apk/*-debug-androidTest-unaligned.apk \
  --device-ids Nexus6\
  --os-version-ids 22 \
  --locales en \
  --orientations portrait \
  --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" \
--directories-to-pull=/sdcard

I am expecting the coverage report to be generated but I get this in the instruments.results file

INSTRUMENTATION_STATUS: numtests=1
INSTRUMENTATION_STATUS: stream=
com.godaddy.gdm.telephony.uitests.DialerTabTest:
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: test=dialerTabNumberFormattingTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: numtests=1
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: test=dialerTabNumberFormattingTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_RESULT: stream=

Time: 6.022

OK (1 test)


Error: **Failed to generate emma coverage.**
INSTRUMENTATION_CODE: -1

And logcat says this:

11-18 21:38:39.400: I/TestRunner(5246): run finished: 1 tests, 0 failed, 0 ignored
11-18 21:38:39.400: I/TestRunner(5246): [ 11-18 21:38:39.400  5246: 5263 E/         ]
11-18 21:38:39.400: I/TestRunner(5246): Failed to generate emma coverage.
11-18 21:38:39.400: I/TestRunner(5246): java.lang.reflect.InvocationTargetException
11-18 21:38:39.400: I/TestRunner(5246):   at java.lang.reflect.Method.invoke(Native Method)
11-18 21:38:39.400: I/TestRunner(5246):   at java.lang.reflect.Method.invoke(Method.java:372)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:80)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:68)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:94)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:69)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
11-18 21:38:39.400: I/TestRunner(5246):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)
11-18 21:38:39.400: I/TestRunner(5246): Caused by: java.io.FileNotFoundException: /sdcard/coverage.ec: open failed: EACCES (Permission denied)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.IoBridge.open(IoBridge.java:456)
11-18 21:38:39.400: I/TestRunner(5246):   at java.io.FileOutputStream.<init>(FileOutputStream.java:87)
11-18 21:38:39.400: I/TestRunner(5246):   at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
11-18 21:38:39.400: I/TestRunner(5246):   ... 8 more
11-18 21:38:39.400: I/TestRunner(5246): Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.Posix.open(Native Method)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.BlockGuardOs.open(BlockGuardOs.java:186)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.IoBridge.open(IoBridge.java:442)
11-18 21:38:39.400: I/TestRunner(5246):   ... 10 more

I can give more information if needed.

How do you get the code coverage file on your local machine? — spierce7, Mar 02 '18 at 22:43
It is usually in the app/build/outputs//jacoco/coverage.ec file. For exact one try a search for `coverage.ec` in your project folder — satyajit, Mar 04 '18 at 02:45
Right, but when you run it with firebase it's not going to be there on your local machine, right? I'm asking how do you get it off of firebase onto your local machine? — spierce7, Mar 04 '18 at 04:05
oh you need gsutil command. Take a look at https://medium.com/mobile-testing/combining-code-coverage-reports-from-ftl-and-unit-tests-9c795ea269da — satyajit, Mar 06 '18 at 21:14

score 15 · Answer 1 · edited May 23 '17 at 11:53

15

It turned out all that needed was to enable the proper permission for Writing on SD card.

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

in AndroidManifest.xml like in this SO answer

Also, make sure testCoverageEnabled true is enabled in app/build.gradle for debug

debug {
            testCoverageEnabled true
}

from this medium post

edited May 23 '17 at 11:53

Community

1
1

answered Jan 17 '17 at 00:28

satyajit

1,470
3
22
43

2

Remember to add this permission just in debug build type if your app doesn't need this for other reason – André Mion Mar 16 '18 at 10:19
8

If you enable the orchestrator using flag `--use-orchestrator`, you should use "coverageFilePath" rather than "coverageFile" to get all the .ec files. Otherwise you will get an exception. check release notes of [AndroidTestOrchestrator 1.0.2-beta1](https://developer.android.com/training/testing/release-notes) And I also didn't get success for a file path other than "/sdcard/coverage.ec" – Islam Salah Jul 17 '18 at 13:08

score 1 · Answer 2 · answered Nov 27 '20 at 09:19

In case somebody meets this on Android 10, on Firebase Test Lab and above answer doesn't work, try adding grant rule (along with permission in Manifest and testCoverageEnabled true).

@get:Rule
val runtimePermissionRule: GrantPermissionRule = GrantPermissionRule.grant(
    Manifest.permission.WRITE_EXTERNAL_STORAGE,     
    Manifest.permission.READ_EXTERNAL_STORAGE
)

How to get code coverage reports from google Firebase for Android Espresso tests

2 Answers2

Linked