Using SoftAssert with Parallel Execution in TestNG produces unexpected results

Question

I am using the rest-assured library to test our REST api that deals with data on sports. In short I have 2 different @Test methods to call per sport, one @Test method to make multiple GET requests to gather all athlete image urls and store in a static ArrayList, and the other method to instantiate a SoftAssert object and actually call all of the url's in a for loop and soft assert a 200 response code. I then do a assertAll() at the end of the 2nd test method.

For example - I have a @Test getSoccerAthletes() which gathers all the urls from the response, the method will repeat until all athlete urls are gathered as the response is limited to 250 athletes at a time. After this method finishes, then the 2nd @Test method for Soccer will execute, it is named testSoccerAthletes() and you can see that it uses dependsOnMethods. Below is the setup.

@Test(priority = 1)
    public static void getSoccerAthletes() {
        baseURI = "https://XXXXX";
        basePath = "XXXXX";

        Response res = given().queryParam("apikey", "XXXXXXXX")
                .queryParam("top", "250")
                .queryParam("skip", soccerSkip)
                .when().get();
        Assert.assertEquals(res.statusCode(), 200);

        JsonPath jPath = res.jsonPath();
        System.out.println("Soccer:  currentPageStart/totalCount " + jPath.getInt("currentPageStart")
                + "/" + jPath.getInt("totalCount"));

        if (soccerSkip == 0) {
            allSoccerAthletes = jPath.get("page.links.headshots.full");
        } else {
            ArrayList<String> athletes = jPath.get("page.links.headshots.full");
            allSoccerAthletes.addAll(athletes);
        }

        if (jPath.getInt("currentPageStart") + 250 < jPath.getInt("totalCount")) {
            soccerSkip += 250;
            getSoccerAthletes();
        }
    }

    @Test(priority = 2, dependsOnMethods = {"getSoccerAthletes"})
    public static void testSoccerAthletes() {
        SoftAssert softAssert = new SoftAssert();
        for (int i = 0; i < allSoccerAthletes.size(); i++) {
            String url = allSoccerAthletes.get(i);
            System.out.println("Soccer Athlete: " + i + "/" + allSoccerAthletes.size());

            Response res = when().head(url);
            softAssert.assertEquals(res.statusCode(), 200, "Failed url: " + url);

            if (i == allSoccerAthletes.size() - 1)
                allSoccerAthletes.clear();
        }
        softAssert.assertAll();
    }

I am seeing varying results, some of which have failures from the @Test testXXXXAthletes methods mixed up. This first suggestion online was to instantiate a SoftAssert object in each @Test method (which I currently am for every testXXXXAthletes method) so that can't be it.

I am starting to lean toward there are thread safety issues, but I am not really sure how to move forward with a solution. Reasons I believe there are thread safety issues are from some articles I have looked into, but do not fully understand. Articles --> https://learn2automate.wordpress.com/2017/07/13/parallel-testng-soft-assertions-the-right-way/, Retrieve test name on TestNG, https://github.com/rest-assured/rest-assured/issues/1420

Any help in resolving my issues would be greatly appreciated! Btw I have also looked into using the @DataProvider annotations and that brings me to other questions about the structure of these tests. The getXXXAthletes methods are acting as data providers to the testXXXAthletes methods, but they have the dependsOnMethods attribute. Should I be using both DataProvider and dependsOnMethods, or one over the other?

score 0 · Accepted Answer · answered Mar 15 '22 at 17:25

Further research confirmed my theory about the thread-safety issues. I was able to conclude that the issue has nothing to do with SoftAssert and everything to do with rest-assured not being thread-safe. That information can be found here --> https://github.com/rest-assured/rest-assured/pull/851

I was able to find a thread-local branch in which someone kindly did the work to make rest-assured thread-safe. I downloaded 2 files RestAssuredThreadLocal and RestAssuredThreadLocalImpl which can be seen here --> https://github.com/rest-assured/rest-assured/commit/3307ba6c79c5547e88cea286d38e5c8a6d679229

After downloading those 2 files, there were some errors that needed resolved and some deprecated methods needed replaced. After that I was able to successfully run my rest-assured tests in parallel with TestNG with the correct results.

Using SoftAssert with Parallel Execution in TestNG produces unexpected results

1 Answers1