48

I'm trying to analyze and fix a flaky test which is often green.
My problem is that once the test passes Bazel doesn't re-run it until any of the inputs have changed.
I saw you can ask Bazel to re-run a target but AFAICT it's only until the first time it's green (i.e. to mitigate a flaky test and not solve it).

Is there a way to ask Bazel to run the test even if it passed?
I'd like something like bazel test --force-attempts=50 //my-package:my-target

Ittai
  • 5,625
  • 14
  • 60
  • 97

2 Answers2

70

There's a flag for it

--cache_test_results=(yes|no|auto) (-t)

If this option is set to 'auto' (the default) then Bazel will only rerun a test if any of the following conditions applies:

  • Bazel detects changes in the test or its dependencies
  • the test is marked as external
  • multiple test runs were requested with --runs_per_test the test failed. If 'no', all tests will be executed unconditionally.

If 'yes', the caching behavior will be the same as auto except that it may cache test failures and test runs with --runs_per_test.

Note that test results are always saved in Bazel's output tree, regardless of whether this option is enabled, so you needn't have used --cache_test_results on the prior run(s) of bazel test in order to get cache hits. The option only affects whether Bazel will use previously saved results, not whether it will save results of the current run.

Users who have enabled this option by default in their .bazelrc file may find the abbreviations -t (on) or -t- (off) convenient for overriding the default on a particular run.

https://docs.bazel.build/versions/master/user-manual.html#flag--cache_test_results

Boris Verkhovskiy
  • 14,854
  • 11
  • 100
  • 103
Ittai
  • 5,625
  • 14
  • 60
  • 97
42

In addition to --cache_test_results, Bazel actually has a flag specifically designed for diagnosing flaky tests: --runs_per_test, which will rerun a test N times and only keep logs from the failing runs:

$ bazel test --runs_per_test=10 :flaker
INFO: Found 1 test target...
FAIL: //:flaker (run 10 of 10) (see /output/testlogs/flaker/test_run_10_of_10.log).
FAIL: //:flaker (run 4 of 10) (see /output/testlogs/flaker/test_run_4_of_10.log).
FAIL: //:flaker (run 5 of 10) (see /output/testlogs/flaker/test_run_5_of_10.log).
FAIL: //:flaker (run 9 of 10) (see /output/testlogs/flaker/test_run_9_of_10.log).
FAIL: //:flaker (run 3 of 10) (see /output/testlogs/flaker/test_run_3_of_10.log).
Target //:flaker up-to-date:
  bazel-bin/flaker
INFO: Elapsed time: 0.828s, Critical Path: 0.42s
//:flaker                                                                FAILED
 
Executed 1 out of 1 tests: 1 fails locally.

You can use it to quickly figure out how flaky a test is and get some failing logs.

Valentin
  • 917
  • 14
  • 19
kris
  • 23,024
  • 10
  • 70
  • 79
  • 6
    One catch with `--runs_per_test` is that bazel runs the tests in parallel. This may not be desirable if the test is accessing a global structure e.g the file system and a proper synchronization is not available. – Pavan Manjunath Jun 27 '19 at 07:06
  • 14
    You can pass `--local_test_jobs=1` if you want to use `--runs_per_test` but not in parallel. – wchargin Mar 25 '20 at 22:33