Artillery.io: How to generate test report for each Scenario?

Question

Artillery: How to run the scenarios sequentially and also display the results of each scenario in the same file?

I'm currently writing nodejs test with artillery.io to compare performance between two endpoints that I implemented. I defined two scenarios and I would like to get the result of each in a same report file. The execution of the tests is not sequential, it means that at the end of the test I have a result already combined and impossible to know the performance of each one but for all.

config:
  target: "http://localhost:8080/api/v1"
  plugins:
    expect: {}
    metrics-by-endpoint: {}
  phases:
    - duration: 60
      arrivalRate: 2
  environments:
    dev:
      target: "https://backend.com/api/v1"
      phases:
        - duration: 60
          arrivalRate: 2
scenarios:
  - name: "Nashhorn"
    flow:
      - post:
          url: "/casting/nashhorn"
          auth:
            user: user1
            pass: user1
          json:
            body:
              fromFile: "./casting-dataset-01-as-input.json"
              options:
                filename: "casting_dataset"
                conentType: "application/json"
          expect:
            statusCode: 200
          capture:
            regexp: '[^]*'
            as: 'result'
      - log: 'result= {{result}}'

  - name: "Nodejs"
    flow:
      - post:
          url: "/casting/nodejs"
          auth:
            user: user1
            pass: user1
          json:
            body:
              fromFile: "./casting-dataset-01-as-input.json"
              options:
                filename: "casting_dataset"
                conentType: "application/json"
          expect:
            statusCode: 200
          capture:
            regexp: '[^]*'
            as: 'result'
      - log: 'result= {{result}}'

How to run the scenarios sequentially and also display the results of each scenario in the same file?

Thank you in advance for your answers

Borjinha10 · Answer 1 · 2020-04-23T07:35:33.710

I think you miss the param weight, this param defines de probability to execute the scenario. if in you first scenario put a weight of 1 and in the second put the same value, both will have the same probability to been execute (50%).

If you put in the first scenario a weight of 3 and in the second one a weight of 1, the second scenario will have a 25% probability of execution while the first one will have a 75% probability of being executed.

This combined with the arrivalRate parameter and setting the value of rampTo to 2, will cause 2 scenarios to be executed every second, in which if you set a weight of 1 to the two scenarios, they will be executed at the same time.

Look down for scenario weights in the documentation

scenarios:
  - flow:
       - log: Scenario for GET requests
       - get: 
            url: /v1/url_test_1
        name: Scenario for GET requests
        weight: 1
  - flow:
       - log: Scenario for POST requets
       - post:
             json: {}
             url: /v1/url_test_2
        name: Scenario for POST
        weight: 1

I hope this helps you.

score 1 · Answer 2 · answered Jan 06 '21 at 18:32

To my knowledge, there isn't a good way to do this with the existing the artillery logic.

using this test script:

scenarios:
   - name: "test 1"
     flow:
     - post:
       url: "/postman-echo.com/get?test=123"
       weight: 1
  - name: "test 2"
    flow:
    - post:
      url: "/postman-echo.com/get?test=123"
      weight: 1
... etc...


Started phase 0 (equal weight), duration: 1s @ 13:21:54(-0500) 2021-01-06
Report @ 13:21:55(-0500) 2021-01-06
Elapsed time: 1 second
  Scenarios launched:  20
  Scenarios completed: 20
  Requests completed:  20
  Mean response/sec: 14.18
  Response time (msec):
    min: 117.2
    max: 146.1
    median: 128.6
    p95: 144.5
    p99: 146.1
  Codes:
    404: 20

All virtual users finished
Summary report @ 13:21:55(-0500) 2021-01-06
  Scenarios launched:  20
  Scenarios completed: 20
  Requests completed:  20
  Mean response/sec: 14.18
  Response time (msec):
    min: 117.2
    max: 146.1
    median: 128.6
    p95: 144.5
    p99: 146.1
  Scenario counts:
    test 7: 4 (20%)
    test 5: 2 (10%)
    test 3: 1 (5%)
    test 1: 4 (20%)
    test 9: 2 (10%)
    test 8: 3 (15%)
    test 10: 2 (10%)
    test 4: 1 (5%)
    test 6: 1 (5%)
  Codes:
    404: 20

So basically you can see that they are weighted equally, but are not running equally. So I think there needs to be something added to the code itself for artillery. Happy to be wrong here.

score 0 · Answer 3 · answered Mar 16 '21 at 08:55

You can use the per endpoint metrics plugin to give you the results per endpoint instead of aggregated.

https://artillery.io/docs/guides/plugins/plugin-metrics-by-endpoint.html

I see you already have this in your config, but it cannot be working if it is not giving you what you need. Did you install it as well as add to config?

npm install artillery-plugin-metrics-by-endpoint

In terms of running sequentially, I'm not sure why you would want to, but assuming you do, you just need to define each POST as part of the same Scenario instead of 2 different scenarios. That way the second step will only execute after the first step has responded. I believe the plugin is per endpoint, not per scenario so will still give you the report you want.

Artillery.io: How to generate test report for each Scenario?

3 Answers3