2

When I run android CTS full test using below command

run cts --plan CTS

Every time it shows different result for some of the packages, I mean some packages some tests passes/fails randomly every time I re-run full test. But when I run package individually (The package in which some tests failed), all the tests passes in it.

Why I am seeing this behavior?

Environment: OS: Android L CTS version: 5.1_r7

SD.
  • 1,432
  • 22
  • 38
  • please provide some detail about tests. – Bhagirathsinh Gohil May 24 '16 at 07:56
  • @DreamCoder, after following all the prerequisite for CTS setup, I ran command as above. It seems that CTS package itself unable to handle some exceptions during full run and that's why some tests in packages are failing and the same tests pass if I run that package individually. – SD. May 31 '16 at 06:46

2 Answers2

0

It happen some time some test failed randomly because some time that test condition is satisfied some time not and some time because of timeout test may fail.

Bhagirathsinh Gohil
  • 673
  • 1
  • 9
  • 25
0

Some cts tests involve specific timeouts set for some event to occur. For example if you are running cts test related to data calls like turning mobile data off/on and timeout to get mobile data connected is set to 10 seconds, then some time this test will pass and sometime it will fail. In this case, increasing that timeout will resolve this issue.

Regarding issue of test case failing when running multiple packages, there could be possibility that test case before failed one has not set device in a neutral/original state for next test. It is a good practice to revert all changes made during a test while exiting a test case.

M.Uzair Afzal
  • 56
  • 7