Firebase Test Lab not honoring coverageFile environment variable

Question

I'm using CircleCi to ask Firebase Test Lab to run instrumented tests for me. The tests all run fine, and as far as I know I've setup everything correctly. However, I really want to get my code coverage.

The SD card never contains a code coverage file and in all the logs I find this exception which I suspect is the issue:

    java.io.FileNotFoundException: /jacoco.exec (Read-only file system)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:287)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:223)
        at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.openFile(FileOutput.java:67)
        at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.startup(FileOutput.java:49)
        at org.jacoco.agent.rt.internal_8ff85ea.Agent.startup(Agent.java:122)
        at org.jacoco.agent.rt.internal_8ff85ea.Agent.getInstance(Agent.java:50)
        at org.jacoco.agent.rt.internal_8ff85ea.Offline.<clinit>(Offline.java:31)
        at org.jacoco.agent.rt.internal_8ff85ea.Offline.getProbes(Offline.java:51)
        at com.acme.android.MyApplication.$jacocoInit(Unknown Source:13)
        at com.acme.android.MyApplication.<clinit>(Unknown Source:0)
        at java.lang.Class.newInstance(Native Method)
        at android.app.Instrumentation.newApplication(Instrumentation.java:1101)
        at android.app.Instrumentation.newApplication(Instrumentation.java:1086)
        at android.app.LoadedApk.makeApplication(LoadedApk.java:942)
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5765)
        at android.app.ActivityThread.-wrap1(Unknown Source:0)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1661)
        at android.os.Handler.dispatchMessage(Handler.java:105)
        at android.os.Looper.loop(Looper.java:164)
        at android.app.ActivityThread.main(ActivityThread.java:6541)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767)

This is how I call Firebase Test Lab:

    - run:
        name: Run instrumented test on Firebase Test Lab
        command: gcloud firebase test android run --type instrumentation --app debug-app.apk --test debug-test.apk --device model=Nexus5X,version=26,locale=en,orientation=portrait --environment-variables coverage=true,coverageFile="/sdcard/tempDir/coverage.ec" --directories-to-pull=/sdcard --timeout 20m --no-record-video

This is how I grab the artifacts that should contain the coverage file:

  - run:
      name: Download instrumented test results from Firebase Test Lab
      command: gsutil -m cp -r -U "`gsutil ls gs://test-lab-XXXXXX-YYYYY | tail -1`*" /root/workspace/firebase/ || true

I've also confirmed that the coverage file isn't stored as an artifact in Firebase.

And my gradle file has code coverage enabled:

buildTypes {
    debug {
        matchingFallbacks = ["prod"]
        minifyEnabled false
        debuggable true
        versionNameSuffix '-DEBUG'
        testCoverageEnabled true
    }

It appears that jacoco is defaulting to the root folder, and is getting permission denied by the Android file system when trying to write the coverage file there. Any help would be greatly appreciated.

Thanks,

Have you check this build.gradle https://github.com/googlesamples/android-architecture-components/blob/master/GithubBrowserSample/app/build.gradle of a google sample app? It has some references to Jacoco test coverage maybe can help you to find what's the problem with your configuration — MatPag, Jan 30 '18 at 22:57
And check [this script too](https://github.com/googlesamples/android-architecture-components/blob/master/test_all_ftl.sh) — MatPag, Jan 30 '18 at 23:02
@MatPag Thanks for passing those alonge. The script your reference does not seem to do anything related to code coverage. The github .gradle file is nearly identical to the one I have. I should probably mention that when I run the instrumented tests locally I can generate code coverage reports just fine. My problem seems to be specific to running in Test Lab. — etherton, Jan 31 '18 at 16:06

score 3 · Answer 1 · answered Apr 10 '18 at 23:17

3

I found the answer here: https://medium.com/mobile-testing/generating-code-coverage-from-espresso-tests-running-in-firebase-test-lab-e98edcea7bdf

The problem was the path for the coverage file. Instead of /sdcard/tempDir/coverage.ec it should be just /sdcard/coverage.ec. Turns out tempDir doesn't exist and thus it fails when trying to write to that folder.

This is the environmental variable options that work:

--environment-variables coverage=true,coverageFile="/sdcard/coverage.ec

answered Apr 10 '18 at 23:17

etherton

924
2
10
21

if use enable orchestrator using `--use-orchestrator` refer to this [comment](https://stackoverflow.com/questions/40726208/how-to-get-code-coverage-reports-from-google-firebase-for-android-espresso-tests#comment89736153_41687205) – Islam Salah Jul 17 '18 at 13:11

Firebase Test Lab not honoring coverageFile environment variable

1 Answers1