I have created iOS Lane for uploading Flutter integration tests to Firebase TestLab and it fails when trying to run the test. When I upload manually ios_tests.zip file which contains Runner_iphoneos folder and Release_iphoneos15.2-arm64.xctestrun, it pass on TestLab, but when using workflow .yml file in github action, TestLab fails to execute the tests.
Here is the error log for specific failing part:
Uploading [build/ios_integ/Build/Products/ios_tests.zip] to Firebase Test Lab...
Test [matrix-1aenoclku3vww] has been created in the Google Cloud.
Firebase Test Lab will execute your xctest test on 1 device(s).
Creating individual test executions...................failed.
ERROR: (gcloud.firebase.test.ios.run)
Matrix [matrix-1aenoclku3vww] failed during validation: The XCTest zip file was malformed. The zip did not contain a single .xctestrun file and the contents of the DerivedData/Build/Products directory..
So the ios_tests.zip file is uploaded and I can find it in GCS bucket. When I extract it I can find all the files. But when I try to manually upload this file to TestLab it gives the same error. Seems like it is somehow changed in the build or upload process. Also it is strange that logs say that Matrix [matrix-1aenoclku3vww] is created, but I cannot see it in firebase console in TestLab. Like its not run at all.
This is the .yml script I use, it fails only at "Run Instrumentation Tests in Firebase Test Lab":
name: Firebase_test
on: [workflow_dispatch]
jobs:
iOS_Lane:
name: iOS Lane
runs-on: macos-latest
env:
PROJECT_DIRECTORY: cicd
defaults:
run:
working-directory: ./${{ env.PROJECT_DIRECTORY }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-java@v2
with:
distribution: "zulu"
java-version: "11"
- uses: subosito/flutter-action@v2
with:
flutter-version: "3.0.2"
- name: Fetching Flutter dependencies
run: flutter pub get
- name: Install the Apple certificate and provisioning profile
env:
BUILD_CERTIFICATE_BASE64: ${{ secrets.BUILD_CERTIFICATE_BASE64 }}
P12_PASSWORD: ${{ secrets.P12_PASSWORD }}
BUILD_PROVISION_PROFILE_BASE64: ${{ secrets.BUILD_PROVISION_PROFILE_BASE64 }}
KEYCHAIN_PASSWORD: ${{ secrets.KEYCHAIN_PASSWORD }}
run: |
# create variables
CERTIFICATE_PATH=$RUNNER_TEMP/build_certificate.p12
PP_PATH=$RUNNER_TEMP/build_pp.mobileprovision
KEYCHAIN_PATH=$RUNNER_TEMP/app-signing.keychain-db
# import certificate and provisioning profile from secrets
echo -n "$BUILD_CERTIFICATE_BASE64" | base64 --decode --output $CERTIFICATE_PATH
echo -n "$BUILD_PROVISION_PROFILE_BASE64" | base64 --decode --output $PP_PATH
# create temporary keychain
security create-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
security set-keychain-settings -lut 21600 $KEYCHAIN_PATH
security unlock-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
# import certificate to keychain
security import $CERTIFICATE_PATH -P "$P12_PASSWORD" -A -t cert -f pkcs12 -k $KEYCHAIN_PATH
security list-keychain -d user -s $KEYCHAIN_PATH
# apply provisioning profile
mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
cp $PP_PATH ~/Library/MobileDevice/Provisioning\ Profiles
- name: Build Firebase debug
run: flutter build ios integration_test/smoke_test.dart --release
- name: Build Firebase build
env:
OUTPUT_PATH: ../build/ios_integ
run: |
pushd ios
xcodebuild clean build -workspace Runner.xcworkspace -scheme Runner -config Flutter/Release.xcconfig -derivedDataPath $OUTPUT_PATH -sdk iphoneos build-for-testing
popd
- name: Zip Firebase build
env:
PRODUCT_PATH: build/ios_integ/Build/Products
DEV_TARGET: 15.2
run: |
pushd $PRODUCT_PATH
zip -r "ios_tests.zip" "Release-iphoneos" "Runner_iphoneos$DEV_TARGET-arm64.xctestrun"
popd
- id: "auth"
uses: "google-github-actions/auth@v0"
with:
credentials_json: "${{ secrets.FIREBASE_SERVICE_JSON }}"
- name: "Set up Cloud SDK"
uses: "google-github-actions/setup-gcloud@v0"
- name: "Use gcloud CLI"
run: "gcloud info"
- name: Set current project
run: gcloud config set project ${{ secrets.GCP_PROJECT_ID }}
- name: Run Instrumentation Tests in Firebase Test Lab
env:
ZIP_LOCATION: build/ios_integ/Build/Products/ios_tests.zip
DEVICE_MODEL: iphone13pro,version=15.2,locale=en_US,orientation=portrait
run: gcloud firebase test ios run --test $ZIP_LOCATION --device model=$DEVICE_MODEL --timeout 3m
Thanks in advance!