Testing my Xamarin.Forms iOS Application with Xamarin.UITest: The first test always fails

Question

I am testing an Xamarin.iOS Application on a real device (iPhone 6s with iOS 12.1) with Xamarin.UITest. When I am running my few UI Tests, the first test always seam to crash (regardless of what is happening inside the test) due to the same error (see below).

Enviroment is:

Xamarin.UITest 2.2.7
NUnitTestAdapter 2.1.1
NUnit 2.6.3
Xamarin.TestCloud.Agent 0.21.7

Setup is:

[SetUp]
public void Setup(){
     this.app = ConfigureApp.iOS
        .EnableLocalScreenshots()
                .InstalledApp("my.app.bundle")
                .DeviceIdentifier("My-iPhone6s-UDID-With-iOS12.1")  
                .StartApp();
}

[Test]
public void FirstTestThatCouldBeEmpty(){
     //But doesn't have to be empty to produce the error    
}

Resulting error:

2019-01-17T14:11:20.4902700Z 1 - ClearData:> 2019-01-17T14:11:20.4916340Z bundleId: my.app.bundle 2019-01-17T14:11:20.4929580Z deviceId: My-iPhone6s-UDID-With-iOS12.1

2019-01-17T14:11:33.7561260Z 3 - LaunchTestAsync:

2019-01-17T14:11:33.7574050Z deviceId: My-iPhone6s-UDID-With-iOS12.1

2019-01-17T14:11:33.9279420Z 5 - HTTP request failed, retry limit hit

2019-01-17T14:11:33.9302300Z Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: Unable to read data from the transport connection: Connection reset by peer. ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer

2019-01-17T14:11:33.9322710Z at System.Net.Sockets.Socket.EndReceive (System.IAsyncResult asyncResult) [0x00012] in <23340a11bb41423aa895298bf881ed68>:0

2019-01-17T14:11:33.9340560Z at System.Net.Sockets.NetworkStream.EndRead (System.IAsyncResult asyncResult) [0x00057] in <23340a11bb41423aa895298bf881ed68>:0

2019-01-17T14:11:33.9358740Z --- End of inner exception stack trace ---

2019-01-17T14:11:33.9377100Z at System.Net.Sockets.NetworkStream.EndRead (System.IAsyncResult asyncResult) [0x0009b] in <23340a11bb41423aa895298bf881ed68>:0

2019-01-17T14:11:33.9398100Z at System.IO.Stream+<>c.b__43_1 (System.IO.Stream stream, System.IAsyncResult asyncResult) [0x00000] in <98fac219bd4e453693d76fda7bd96ab0>:0

2019-01-17T14:11:33.9415720Z at System.Threading.Tasks.TaskFactory1+FromAsyncTrimPromise1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) [0x00000] in <98fac219bd4e453693d76fda7bd96ab0>:0

Sometimes it is this error:

SetUp : Xamarin.UITest.XDB.Exceptions.DeviceAgentException : Unable to end session: An error occurred while sending the request

System.Net.Http.HttpRequestException : An error occurred while sending the request

System.Net.WebException : Unable to read data from the transport connection: Connection reset by peer.

System.IO.IOException : Unable to read data from the transport connection: Connection reset by peer.

System.Net.Sockets.SocketException : Connection reset by peer

What could be a solution to this? Doing the version puzzle of the nuget packages got me this far, that all tests are working besides this one.

A dummy test would be a possibility, but that would that countinous builds with those test would never be in the state of "successful" because this one test that is failing =(

This might seem strange, but have you tried to restart the device and run it again? — Bruno Caceiro, Jan 17 '19 at 15:43
This may be the same issue as this one: https://stackoverflow.com/questions/50024536/xamarin-uitest-2-2-4-start-session-failed-on-the-first-test — Leszek L, Jan 18 '19 at 07:25
Yes, I checked that thread out, but the solution in the thread you mentioned is for simulators. And I am testing on a physical device. So that doesn't help me unfortunately. — Tim S., Jan 18 '19 at 08:09
This is occurring for me as well using Xamarin UI Test 3.0.0. I haven't found a work-around yet. Others please also note, in addition to the hardware difference, this is "connection reset by peer" not "connection refused" — Jessica Pennell, Jul 23 '19 at 17:50
I was able to obtain a more detailed log by saving an exception from TestFixtureSetup and throwing it during Setup. I will omit the part you already pasted. https://pastebin.com/etgzQZLZ . Note that this doesn't always fail during DeleteSessionAsync (e.g. if you manually restart the app under test) but does always fail somewhere inside LaunchTestAsync. — Jessica Pennell, Jul 23 '19 at 18:11

Jessica Pennell · Accepted Answer · 2019-08-01T23:30:14.377

I was able to find a work around for this issue, and really any issue with the device agent not connecting to the app under test.

        public static IApp StartApp(Platform platform)
        {
            if (platform == Platform.iOS)
            {
                try
                {
                    return ConfigureApp.iOS
                       .InstalledApp("com.example")
#if DEBUG
                       .Debug()
#endif
                       .StartApp();
                }
                catch
                {
                    return ConfigureApp.iOS
                       .InstalledApp("com.example")
#if DEBUG
                        .Debug()
#endif
                       .ConnectToApp();
                }
            }

In plain language : "If you tried to start the app, and something went wrong, try to see if the app under test was started anyway and connect to it"

Note : As https://stackoverflow.com/users/1214857/iupchris10 mentioned in the comments, a caveat to this approach is that when you run tests consecutively, while the StartApp call will in theory reliably shut down the app, with the point of failure being reconnection, if it does not and you are running tests consecutively, you will simply connect to the running app instance in whatever state it was left in. This approach offers no recourse. Xamarin's StartApp will throw Exception, rather than a typed exception, and so mitigating this will rely on you parsing InnerException.Message and keeping a table of expected failures.

A side effect of this approach, is that the app won't be relaunched - you'll just keep connecting to the app already running in the foreground. So any test that occurs after the 1st test, may not be on the correct screen it expects to be on. — iupchris10, Jul 31 '19 at 17:40
@iupchris10 Thank you :) That's worth adding to the post proper so people don't have to dig into the comments, thanks for that. E. Which I have just done. StartApp does stop the running instance if it works but... we're swallowing errors so we have no idea whether or not that's really working, and you are completely correct about what happens if it doesn't. — Jessica Pennell, Aug 01 '19 at 23:21

Testing my Xamarin.Forms iOS Application with Xamarin.UITest: The first test always fails

1 Answers1