TFS 2018 and AppCenter Test

Question

I'm new to CI/CD and can't find any information on the internet on how to use AppCenter Test with TFS 2018. I have my Xamarin projects in TFS but I want to test them within the device cloud of AppCenter Test.

There are a couple documents on MS's docs that make it look like it's possible but for a newbie they give ZERO information on how to implement it. HERE is an example. If you look up top this document is relavant to TFS 2018.

Answer 1

In Azure Pipeline, you can use either YAML or the classic editor to define your pipelines. But YAML build is not supported in TFS 2018 yet.

So you need to use the classic editor to add App Center Test task to your pipeline and then refer to the Arguments in the documentation to define the task.

Answer 2

Here is what I did to get it working. No idea if it's correct or not, but hey, it works!

Xamarin.UITest

If you don't do the following step you'll run your tests locally and nothing will happen, they'll just immediately exit.

First go to Tools > Options > Test > General > Active Solution then uncheck For improved performance, only use test adapters in test assembly folder or as specified in runsettings file

public class AppInitializer
    {
        public static IApp StartApp(Platform platform)
        {
            if (platform == Platform.Android)
            {
                return ConfigureApp.Android
                    // You may need to adjust the following relative path based on where you created your UITest project
                    .ApkFile(@"..\..\..\<AppName>\<AppName>.Android\bin\Release\<App Package Name>.apk")
                    // Uncomment if you are running locally and you want VS to launch/install app
                    //.PreferIdeSettings()
                    .StartApp();
            }
            return ConfigureApp.iOS.StartApp();
        }
    }

Do not add references to your Android and iOS projects like the MS docs say. This will lead you down the path of hours of useless troubleshooting

Deploy your agent on a development windows box. Follow these instructions: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops

TFS 2018 Configuration

1. In TFS click into the repository of your project
2. Click "Build and Release"
3. Up top right click "+New"
4. Click "Continue"
5. Scroll down and click "Xamarin.Android"
6. on the next screen set the Agent queue to whatever you created ealier, probably Default

List of my build tasks

• Nuget Tool Installer
• .NET Core
• Nuget
• Xamarin.Android
• MSBuild
• Android signing
• Node Tool Installer
• Nuget
• App Center Test

Here are my build steps in detail

Nuget Tool Installer

• Use Nuget
• version of nuget to install : 4.9.3 (this is what I noticed my VS was using)
• all else defaults

.NET Core

• command: restore
• path to projects: **/*.csproj
• all else defaults

Nuget

• Command: custom
• Command and arguments: restore -MsbuildPath "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin"
• All else defaults

Xamarin.Android

• Project: **/*Droid*.csproj
• Target: (empty)
• Output directory: $(build.binariesdirectory)\$(BuildConfiguration)
• Configuration: $(BuildConfiguration)
• MSBuild: enable Specify Location
• MSBuild Location: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin
• JDK Options, JDK 8 x64
• all else defaults

MSBuild

• Project: (fit this to match your uitest project): **/*UITest*.csproj
• MSBuild version: Latest
• Platform: (empty)
• Configuration: $(BuildConfiguration)
• MSBuild Arguments: /p:OutputPath="$(build.binariesdirectory)\$(BuildConfiguration)\test-assembly"
• all else defaults

Android Signing

• APK Files: $(build.binariesdirectory)\$(BuildConfiguration)\*.apk
• Check sign the APK and put in your info
• Jarsigner arguments: -verbose -sigalg MD5withRSA -digestalg SHA1
• Check to Zipalign
• Zipalign location: (empty)
• all else defaults

Node Tool Installer

• Version Spec: (choose whatever version you already have on your system) 10.11.0
• all else defaults

Note: Prior to the next step create a new variable called XamarinUITestVer and set the value to whatever value you want to use for the Xamarin.UITest Nuget package you want to install. In my case the value I set was: 2.2.7

Nuget

• Command: custom
• Command and Arguments (refer to note above): install Xamarin.UITest -Version $(XamarinUITestVer) -OutputDirectory "$(Agent.BuildDirectory)\Nuget"
• all else defaults

App Center Test

• Binary Application File Path: $(build.binariesdirectory)\$(BuildConfiguration)\*.apk
• Artifacts Directory: $(Build.ArtifactStagingDirectory)\AppCenterTest
• Check: 'Prepare Tests`
• Test Framework: Xamarin UI Test
• Build Directory: $(build.binariesdirectory)\$(BuildConfiguration)\test-assembly
• Store File: (empty)
• Store Password: (empty)
• Key Alias: (empty)
• Key Password: (empty)
• Test Tools Directory: $(Agent.BuildDirectory)\Nuget\Xamarin.UITest.$(XamarinUITestVer)\tools\
• Signing Information: (empty)
• Additional Options: (empty)
• Check: Run Tests
• Authentication Method: App Center Connection
• App Center Connection: (create a new connection)
• App Slug: obtain this by "almost" creating a test in AppCenter. From your app in AppCenter create a New Test Run, select device(s), select Next and choose Xamarin.UITest and hit Next. Down below you'll see the --app string and this is what you use for this field.
• Devices: obtain this by "almost" creating a test in AppCenter. From your app in AppCenter create a New Test Run, select device(s), select Next and choose Xamarin.UITest and hit Next. Down below you'll see the --devices string and this is what you use for this field.
• all else defaults