1

I have created iOS Lane for uploading Flutter integration tests to Firebase TestLab and it fails when trying to run the test. When I upload manually ios_tests.zip file which contains Runner_iphoneos folder and Release_iphoneos15.2-arm64.xctestrun, it pass on TestLab, but when using workflow .yml file in github action, TestLab fails to execute the tests.

Here is the error log for specific failing part:

Uploading [build/ios_integ/Build/Products/ios_tests.zip] to Firebase Test Lab...

Test [matrix-1aenoclku3vww] has been created in the Google Cloud.

Firebase Test Lab will execute your xctest test on 1 device(s).

Creating individual test executions...................failed.

ERROR: (gcloud.firebase.test.ios.run) 

Matrix [matrix-1aenoclku3vww] failed during validation: The XCTest zip file was malformed. The zip did not contain a single .xctestrun file and the contents of the DerivedData/Build/Products directory..

So the ios_tests.zip file is uploaded and I can find it in GCS bucket. When I extract it I can find all the files. But when I try to manually upload this file to TestLab it gives the same error. Seems like it is somehow changed in the build or upload process. Also it is strange that logs say that Matrix [matrix-1aenoclku3vww] is created, but I cannot see it in firebase console in TestLab. Like its not run at all.

This is the .yml script I use, it fails only at "Run Instrumentation Tests in Firebase Test Lab":

name: Firebase_test
on: [workflow_dispatch]
jobs:
  iOS_Lane:
    name: iOS Lane
    runs-on: macos-latest
    env:
      PROJECT_DIRECTORY: cicd
    defaults:
      run:
        working-directory: ./${{ env.PROJECT_DIRECTORY }}
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v2
        with:
          distribution: "zulu"
          java-version: "11"
      - uses: subosito/flutter-action@v2
        with:
          flutter-version: "3.0.2"

      - name: Fetching Flutter dependencies
        run: flutter pub get

      - name: Install the Apple certificate and provisioning profile
        env:
          BUILD_CERTIFICATE_BASE64: ${{ secrets.BUILD_CERTIFICATE_BASE64 }}
          P12_PASSWORD: ${{ secrets.P12_PASSWORD }}
          BUILD_PROVISION_PROFILE_BASE64: ${{ secrets.BUILD_PROVISION_PROFILE_BASE64 }}
          KEYCHAIN_PASSWORD: ${{ secrets.KEYCHAIN_PASSWORD }}
        run: |
          # create variables
          CERTIFICATE_PATH=$RUNNER_TEMP/build_certificate.p12
          PP_PATH=$RUNNER_TEMP/build_pp.mobileprovision
          KEYCHAIN_PATH=$RUNNER_TEMP/app-signing.keychain-db
          # import certificate and provisioning profile from secrets
          echo -n "$BUILD_CERTIFICATE_BASE64" | base64 --decode --output $CERTIFICATE_PATH
          echo -n "$BUILD_PROVISION_PROFILE_BASE64" | base64 --decode --output $PP_PATH
          # create temporary keychain
          security create-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
          security set-keychain-settings -lut 21600 $KEYCHAIN_PATH
          security unlock-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
          # import certificate to keychain
          security import $CERTIFICATE_PATH -P "$P12_PASSWORD" -A -t cert -f pkcs12 -k $KEYCHAIN_PATH
          security list-keychain -d user -s $KEYCHAIN_PATH
          # apply provisioning profile
          mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
          cp $PP_PATH ~/Library/MobileDevice/Provisioning\ Profiles

      - name: Build Firebase debug
        run: flutter build ios integration_test/smoke_test.dart --release

      - name: Build Firebase build
        env:
          OUTPUT_PATH: ../build/ios_integ
        run: |
          pushd ios
          xcodebuild clean build -workspace Runner.xcworkspace -scheme Runner -config Flutter/Release.xcconfig -derivedDataPath $OUTPUT_PATH -sdk iphoneos build-for-testing
          popd
      - name: Zip Firebase build
        env:
          PRODUCT_PATH: build/ios_integ/Build/Products
          DEV_TARGET: 15.2
        run: |
          pushd $PRODUCT_PATH
          zip -r "ios_tests.zip" "Release-iphoneos" "Runner_iphoneos$DEV_TARGET-arm64.xctestrun"
          popd
      - id: "auth"
        uses: "google-github-actions/auth@v0"
        with:
          credentials_json: "${{ secrets.FIREBASE_SERVICE_JSON }}"

      - name: "Set up Cloud SDK"
        uses: "google-github-actions/setup-gcloud@v0"

      - name: "Use gcloud CLI"
        run: "gcloud info"

      - name: Set current project
        run: gcloud config set project ${{ secrets.GCP_PROJECT_ID }}

      - name: Run Instrumentation Tests in Firebase Test Lab
        env:
          ZIP_LOCATION: build/ios_integ/Build/Products/ios_tests.zip
          DEVICE_MODEL: iphone13pro,version=15.2,locale=en_US,orientation=portrait
        run: gcloud firebase test ios run --test $ZIP_LOCATION --device model=$DEVICE_MODEL --timeout 3m

Thanks in advance!

Mile Dev
  • 652
  • 6
  • 13
  • Do you have a `Release-iphoneos` folder and a `Runner_iphoneos15.2-arm64.xctestrun` inside your `ios_tests.zip`? – Andre Oliveira Sep 17 '22 at 09:35
  • 1
    Hello Andre, Yes, I have both Release-iphoneos folder and Runner_iphoneos15.2-arm64.xtestrun file in ios_tests.zip. What I discovered is that there is a difference in local ios_tests.zip file and ios_tests.zip after it is uploaded and stored in GCS bucket. In local ios_tests.zip I can see RunnerTests.xctest file in Release-iphoneos/Runner/Plugins. But after upload, this ios_tests.zip does not contain RunnerTests.xctest file inside Release-iphoneos/Runner/Plugins. And this is the reason why it fails. I dont know where it may be gone or lost.. Still not solved this issue. – Mile Dev Sep 18 '22 at 11:48
  • SOLVED: So after many trials and errors, finally I found what was causing the issue in my pipeline not to upload Release_iphoneos15.2-arm64.xctestrun file to gcloud: Build step in the pipeline which is using this command: `xcodebuild clean build -workspace Runner.xcworkspace -scheme Runner -config Flutter/Release.xcconfig -derivedDataPath $OUTPUT_PATH -sdk iphoneos build-for-testing` is using the latest ios sdk which is iOS 16. So when zipping the file I had to put 16.0 value to variable DEV_TARGET – Mile Dev Jan 17 '23 at 15:20

0 Answers0