Can not setup to execute some scenarios sequentially and some in parallel with tags and thread count in KarateDSL

Question

I have 2 feature files each with 2 scenarios. I want to execute scenarios from: * [First feature], [Second feature] - sequentially; * [Third feature] - in parallel;

I use tags and 2 Test Runners (with threadCount=1, threadCount=5), but all scenarios are executing in parallel (due to timeline.html).

Scenarios:

@sequential
Feature: First feature
  Background:
    # some code

  @smoke
  Scenario: f1, s1
    # some code

  @regression
  Scenario: f1, s2
    # some code

@sequential @smoke
Feature: Second feature
  Background:
    # some code

  @smoke
  Scenario: f2, s1
    # some code

  @regression
  Scenario: f2, s2
    # some code

Feature: Third feature
  Background:
    # some code

  @smoke
  Scenario: f3, s1
    # some code

  @regression
  Scenario: f3, s2
    # some code

Test runners for scenarios that should be executed sequentially:

@KarateOptions(tags = {"@regression,@smoke", "@sequential,~@ignore"})
public class TestsRunner extends TestSetBase {

    @Test
    public void AllSequentialTests() {
        Results results = Runner.parallel(getClass(), 1, PropertyValues.getReportDir());
        generateReport(PropertyValues.getReportDir());
        assertTrue(results.getFailCount() == 0, results.getErrorMessages());
    }
}

Test runner for parallel execution:

@KarateOptions(tags = {"@regression,@smoke", "~@sequential,~@ignore"})
public class TestsRunner2 extends TestSetBase {

    @Test
    public void AllSequentialTests() {
        Results results = Runner.parallel(getClass(), 5, PropertyValues.getReportDir());
        generateReport(PropertyValues.getReportDir());
        assertTrue(results.getFailCount() == 0, results.getErrorMessages());
    }
}

I'm running tests from maven:

...
 <configuration>
      <includes>
          <include>api/TestsRunner.java</include>
          <include>api/TestsRunner2.java</include>
      </includes>
 </configuration>
...

Currently, when I execute tests, timeline.html shows:

---

|ForkJoinPool-2-Worker1|----[f1, s1]

|ForkJoinPool-2-Worker2|----[f1, s2]

|ForkJoinPool-2-Worker3|----[f2, s1]---[f3, s2]

|ForkJoinPool-2-Worker4|----[f2, s2]

|ForkJoinPool-2-Worker5|----[f2, s1]

---

I tried to add @parallel=false, in such case s1, s2 will be executed sequentially, but f1, f2 have still be executed in parallel. I'm using KarateDSL v.0.9.3.

Do you have any ideas how to fix it?

Answer 1

Morning is wiser than evening. I found the mistake.

I assume that:

• SequentialTestRunner {"@regression,@smoke", "@sequential,~@ignore"} - will run all sequential tests;

• ParallelTestRunner {"@regression,@smoke", "~@sequential,~@ignore"} - will run all other (not tagged as sequential) tests in parallel;

... but ParallelTestRunner executes all tests. Therefore when I call both executors from maven, results should look like: all sequential test + all tests (sequential tests will execute second time). ... but as I see executor is quite smart and does not execute sequential tests second time. As a result only ParallelTestRunner results will be shown.

Classical situation when 2 or more wrong assumptions interlace and hide the real problem. Why actually I did not found this issue before.

Therefore it's not an issue, it's just my mistaken assumptions.

Answer 2

I read your question multiple times but gave up. Also I have to say that you may be the first person I have encountered who finds it a problem that scenarios are executing in parallel.

If you want 2 scenarios to be executed back-to-back - in my opinion this is a classic case of wrong test design, where you have a test depending on the other. The right thing to do is to combine both in to one Scenario. You can also try the option (not recommended) of creating a new Scenario and calling these 2 Scenario-s in sequence. Please refer this: https://github.com/intuit/karate#call-tag-selector

If you still feel that something is not working the way you expect, please follow this process, it will be much easier for us to figure out instead of reading though your example: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue