How to do sanity test for a JMeter test suite to ensure tests are not broken?

Question

I have a number of JMeter tests (JMX) and need to ensure they are not broken as I do changes to the utility classes and configuration changes.

User count, loop count, duration, etc are read from JMeter properties (i.e. user.properties).

The variable names in each thread group are different at the moment.

I am using default Thread Groups and Concurrency Thread Groups with throughput shaping timers.

Requirement Each JMX should run with a pre-defined number of iterations and user counts and generate a report with the status.

Is there a way to enforce validation for the entire test plan as in the Thread Group Validation feature through the command line?

Answer 1

Here is my solution to the problem This will run the tests with a given hold target rate and load_profile. The test report is generated at the end and open the report in a browser.

Code snippet is available through Gist

JMETER_HOME=../install/apache-jmeter-5.4.1
TEST_PLAN_HOME=/Users/hansi/Documents/learning/Xap.PerformanceTesting/test_plans/xap/sanity/
TEST_RESULT_HOME=/Users/hansi/Documents/learning/Xap.PerformanceTesting/test_results/sanity/
TEST_REPORTS_HOME=/Users/hansi/Documents/learning/Xap.PerformanceTesting/test_reports/sanity/
TEST_RESULT_FILENAME="test-results-xap-sanity-test"

printf "Starting the sanity test\n"

while :; do
  read -r -p "Please enter hold rate in minutes (1-10) " hold_target_rate_in_min
  hold_target_rate_in_min=${hold_target_rate_in_min:-1}
  [[ $hold_target_rate_in_min =~ ^[0-9]+$ ]] || { echo "Enter a valid number"; continue; }
  if ((hold_target_rate_in_min >= 1 && hold_target_rate_in_min <= 10)); then
    echo "Valid number of hold_target_rate_in_min"
    break
  else
    echo "number $hold_target_rate_in_min is out of range, try again"
  fi
done


mkdir -pv $TEST_RESULT_HOME
mkdir -pv $TEST_REPORTS_HOME
>"$TEST_RESULT_HOME"sucess-sanity-test.log
>"$TEST_RESULT_HOME"error-sanity-test.log


timestamp=$(date +"%y-%b-%d-%H-%M")
TEST_RESULT_FILENAME="$TEST_RESULT_FILENAME$timestamp".csv
TEST_RESULT_FILE_FULL_PATH=$TEST_RESULT_HOME$TEST_RESULT_FILENAME
cd $JMETER_HOME/bin || exit

test -f  "$TEST_RESULT_FILE_FULL_PATH" && rm "$TEST_RESULT_FILE_FULL_PATH"

fileCountSuccess=0
fileCountFailed=0
for testPlan in $TEST_PLAN_HOME/*.jmx; do


    printf "==> Started : %s \a\n\n"  `basename "$testPlan"`
    if ./jmeter.sh -n -t "$testPlan"  \
                -Jhold_target_rate_in_min="$hold_target_rate_in_min" \
                -Jload_profile="line(1,5,1m) const(5,3m) line(5,1,1m) " \
                -l "$TEST_RESULT_FILE_FULL_PATH";
    then
        fileCountSuccess=$((fileCountSuccess+1))
        printf "Test plan %s was completed successfully\n" `basename "$testPlan"` >> "$TEST_RESULT_HOME"sucess-sanity-test.log

    else
        printf "Test plan %s was failed \n" `basename "$testPlan"` >> "$TEST_RESULT_HOME"error-sanity-test.log
        fileCountFailed=$((fileCountFailed+1))
    fi

    printf "==> Completed : %s  \n\n\a" `basename "$testPlan"`
done

if [ -f "$TEST_RESULT_FILE_FULL_PATH" ];then
 ./jmeter.sh -g "$TEST_RESULT_FILE_FULL_PATH" -f -o $TEST_REPORTS_HOME
fi


printf "Process is completed!\n Successful test plans : %s \n Failed Test Plans %s\n" $fileCountSuccess $fileCountFailed
test -f $TEST_REPORTS_HOME/index.html && open $TEST_REPORTS_HOME/index.html

References:

1. Checking input is a valid number within a range
2. Getting the latest version of the JMX within subfolders
3. Open a file in the default browser
4. Adding a timestamp
5. How to create a symbolic link
6. Why printf is better than echo
7. Loop through files in a directory

Answer 2

In normal projects it's being done by Gitflow and Continuous Integration

1. You make your changes in a branch
2. When you're done and you want to see if it still works you create a pull request to the master branch
3. At this stage a Continuous Integration server runs your JMeter test
4. If it passes you will get some form of +1 response from the Continuous Integration server and it will be able to submit your changes to the master branch with the confidence that everything will still work as it used to
5. If it fails you will get notified and won't be able to submit your changes

It also allows you to track changes history, names of people who added/removed/edited this or that line, etc.