Behave: Writing a Scenario Outline with dynamic examples

Question

Gherkin / Behave `Examples`

Gherkin syntax features test automation using examples:

Feature: Scenario Outline (tutorial04)

  Scenario Outline: Use Blender with <thing>
    Given I put "<thing>" in a blender
    When I switch the blender on
    Then it should transform into "<other thing>"

    Examples: Amphibians
        | thing         | other thing |
        | Red Tree Frog | mush        |
        | apples        | apple juice |

    Examples: Consumer Electronics
        | thing         | other thing |
        | iPhone        | toxic waste |
        | Galaxy Nexus  | toxic waste |

The test suite would run four times, once for each example, giving a result similar to:

My problem

How can I test using confidential data in the Examples section? For example, I would like to test an internal API with user ids or SSN numbers, without keeping the data hard coded in the feature file.

Is there a way to load the Examples dynamically from an external source?

Update: Opened a github issue on the behave project.

This change would affect the Gherkin language itself. If it is implemented for 'behave' only then behave would start using a dialect of Gherkin that is vastly different from the Gherkin that is currently used by cucumber-ruby, cucumber-java, specflow and who knows what else. You should try to make it possible that fake data can be used with your code, data that you can safely stored in the feature file. — Szabo Peter, Jan 11 '17 at 10:37

score 5 · Answer 1 · answered Apr 06 '21 at 21:45

I've come up with another solution (behave-1.2.6):

I managed to dynamically create examples for a Scenario Outline by using before_feature.

Given a feature file (x.feature):

Feature: Verify squared numbers

  Scenario Outline: Verify square for <number>
    Then the <number> squared is <result>

Examples: Static
  | number | result |
  |   1    |    1   | 
  |   2    |    4   |
  |   3    |    9   |
  |   4    |   16   |

  # Use the tag to mark this outline
  @dynamic
  Scenario Outline: Verify square for <number>
    Then the <number> squared is <result>

Examples: Dynamic
  | number | result |
  |   .    |    .   |

And the steps file (steps/x.step):

from behave import step

@step('the {number:d} squared is {result:d}')
def step_impl(context, number, result):
    assert number*number == result

The trick is to use before_feature in environment.py as it has already parsed the examples tables to the scenario outlines, but hasn't generated the scenarios from the outline yet.

import behave
import copy

def before_feature(context, feature):
    features = (s for s in feature.scenarios if type(s) == behave.model.ScenarioOutline and
                'dynamic' in s.tags)
    for s in features:
        for e in s.examples:
            orig = copy.deepcopy(e.table.rows[0])
            e.table.rows = []
            for num in range(1,5):
                n = copy.deepcopy(orig)
                # This relies on knowing that the table has two rows.
                n.cells = ['{}'.format(num), '{}'.format(num*num)]
                e.table.rows.append(n)

This will only operate on Scenario Outlines that are tagged with @dynamic.

The result is:

behave -k --no-capture
Feature: Verify squared numbers # features/x.feature:1

  Scenario Outline: Verify square for 1 -- @1.1 Static  # features/x.feature:8
    Then the 1 squared is 1                             # features/steps/x.py:3

  Scenario Outline: Verify square for 2 -- @1.2 Static  # features/x.feature:9
    Then the 2 squared is 4                             # features/steps/x.py:3

  Scenario Outline: Verify square for 3 -- @1.3 Static  # features/x.feature:10
    Then the 3 squared is 9                             # features/steps/x.py:3

  Scenario Outline: Verify square for 4 -- @1.4 Static  # features/x.feature:11
    Then the 4 squared is 16                            # features/steps/x.py:3

  @dynamic
  Scenario Outline: Verify square for 1 -- @1.1 Dynamic  # features/x.feature:19
    Then the 1 squared is 1                              # features/steps/x.py:3

  @dynamic
  Scenario Outline: Verify square for 2 -- @1.2 Dynamic  # features/x.feature:19
    Then the 2 squared is 4                              # features/steps/x.py:3

  @dynamic
  Scenario Outline: Verify square for 3 -- @1.3 Dynamic  # features/x.feature:19
    Then the 3 squared is 9                              # features/steps/x.py:3

  @dynamic
  Scenario Outline: Verify square for 4 -- @1.4 Dynamic  # features/x.feature:19
    Then the 4 squared is 16                             # features/steps/x.py:3

1 feature passed, 0 failed, 0 skipped
8 scenarios passed, 0 failed, 0 skipped
8 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m0.005s

This relies on having an Examples table with the correct shape as the final table, in my example, with two rows. I also don't fuss with creating new behave.model.Row objects, I just copy the one from the table and update it. For extra ugliness, if you're using a file, you can put the file name in the Examples table.

score 4 · Answer 2 · answered Apr 10 '17 at 11:07

Got here looking for something else, but since I've been in similar situation with Cucumber before, maybe someone will also end up at this question, looking for a possible solution. My approach to this problem is to use BDD variables that I can later handle at runtime in my step_definitions. In my python code I can check what is the value of the Gherkin variable and map it to what's needed.

For this example:

Scenario Outline: Use Blender with <thing>
     Given I put "<thing>" in a blender
     When I switch the blender on
     Then it should transform into "<other thing>"

    Examples: Amphibians
        | thing         | other thing            |
        | Red Tree Frog | mush                   |
        | iPhone        | data.iPhone.secret_key | # can use .yaml syntax here as well

Would translate to such step_def code:

@given('I put "{thing}" in a blender')
def step_then_should_transform_into(context, other_thing):   
  if other_thing == BddVariablesEnum.SECRET_KEY: 
    basic_actions.load_secrets(context, key)

So all you have to do is to have well defined DSL layer.

score 1 · Answer 3 · answered Apr 11 '17 at 10:40

Regarding the issue of using SSN numbers in testing, I'd just use fake SSNs and not worry that I'm leaking people's private information.

Ok, but what about the larger issue? You want to use a scenario outline with examples that you cannot put in your feature file. Whenever I've run into this problem what I did was to give a description of the data I need and let the step implementation either create the actual data set used for testing or fetch the data set from an existing test database.

Scenario Outline: Accessing the admin interface
  Given a user who <status> an admin has logged in
  Then the user <ability> see the admin interface

Examples: Users
  | status | ability |
  | is     | can     |
  | is not | cannot  |

There's no need to show any details about the user in the feature file. The step implementation is responsible for either creating or fetching the appropriate type of user depending on the value of status.

^ This. For my company at least, you should never use real customer private information in a test. — beer geek, Apr 12 '22 at 16:35

Behave: Writing a Scenario Outline with dynamic examples

Gherkin / Behave `Examples`

My problem

3 Answers3

Linked

Behave: Writing a Scenario Outline with dynamic examples

Gherkin / Behave Examples

My problem

3 Answers3

Linked

Gherkin / Behave `Examples`