0

I am facing intermittent protractor test failures across a range of test suites without any real pattern in the fail cases to indicate what could be going on, for example it's not the same tests that are failing. Sometimes I will get many failures and on other occasions just a single fail case.

I should point out that this tends to only happen when performing test runs on a Jenkins CI server we have configured (running under linux). Locally on Windows dev machines we may get a single fail case after 30-40 runs which I can live with!

The application we are testing is currently built with angular 1.5 and we are using angular material 1.1.3

Due to the animations used in angular material and the performance hit these can have, we have already tried disabling animations following this approach here which certainly make the tests quicker but dont help with the fail cases we are seeing/

I am at a point now where I am running one test suite over and over, after 5 successful runs it then failed on it's 6th attempt on our Jenkins CI environment\linux box, locally I have run this test many times now and no failures yet.

The test suite in question is detailed below along with a page object file snippet:

//test suite
describe('Operators View', function () {
  var operatorPage = require('./operators.po.js'),
    loginView = require('../login/login.po.js'),
    page = new operatorPage();

  describe('Large screen tests', function () {
    beforeAll(function () {
      loginView.login();
    });

    afterAll(function () {
      loginView.logout();
    });

    it('should create an operator', function () {
      page.settlementBtn.click();
      page.operatorsBtn.click();
      page.fabBtn.click();
      page.createOperator();
      expect(page.headline.getText()).toEqual('Operators');
    });
    });
});

// operators.po.js
var operatorsSection = function() {
    this.helper = new Helpers();
    this.headline = element(by.css('.md-headline'));
    this.settlementBtn = element(by.css('[ui-sref="settlement"]'));
    this.operatorsBtn = element(by.css('[ui-sref="operators"]'));
    this.fabBtn = element(by.css('.md-fab'));

    // Form Elements
    this.licenceNumber = element(by.model('vm.transportOperator.licenceNumber'));
    this.tradingName = element(by.model('vm.tradingName'));
    this.name = element(by.model('vm.name'));
    this.operatorAddressFirstLine = element(by.model('vm.transportOperator.address.line1'));
    this.operatorAddressCityTown = element(by.model('vm.transportOperator.address.line5'));
    this.operatorAddressPostCode = element(by.model('vm.transportOperator.address.postcode'));
    this.payeeAddressFirstLine = element(by.model('vm.transportOperator.payee.address.line1'));
    this.payeeAddressCityTown = element(by.model('vm.transportOperator.payee.address.line4'));
    this.payeeAddressPostCode = element(by.model('vm.transportOperator.payee.address.postcode'));
    this.opID = element(by.model('vm.transportOperator.fields.opID'));
    this.spID = element(by.model('vm.transportOperator.fields.spID'));
    this.schemeSelect = element(by.model('reference.scheme'));
    this.schemeOptions = element(by.exactRepeater('scheme in vm.schemes').row('0'));
    this.alias = element(by.model('reference.alias'));
    this.reference = element(by.model('reference.reference'));
    this.saveBtn = element(by.css('.md-raised'));


    this.createOperator = function() {
      this.licenceNumber.sendKeys(this.helper.getRandomId(10));
      this.tradingName.sendKeys('Protractor Trade Name LTD');
      this.name.sendKeys('Protractor Trade Name');
      this.operatorAddressFirstLine.sendKeys('Protractor Town');
      this.operatorAddressCityTown.sendKeys('Cardiff');
      this.operatorAddressPostCode.sendKeys('PT4 4TP');
      this.payeeAddressFirstLine.sendKeys('Protractor Town');
      this.payeeAddressCityTown.sendKeys('Cardiff');
      this.payeeAddressPostCode.sendKeys('PT4 4TP');
      this.opID.sendKeys('177');
      this.spID.sendKeys('Protractor Spid');
      this.schemeSelect.click();
      this.schemeOptions.click();
      this.alias.sendKeys('PTAlias');
      this.reference.sendKeys('Protractor');
      this.saveBtn.click();
    }

  };
module.exports = operatorsSection;

In this test suite after the call to createOperator from the PO file is invoked and the savteBtn is clicked, the application will transition to a state that shows a table of created entries (after successful creation of course). We are using angular ui-router also, currently on version 0.2.18

The expectation fails with:

Expected 'Create An Operator' to equal 'Operators'.

Yet the accompanying screenshot that was captured shows the table view with an 'Operators' heading, it seems the call to page.headline.getText() inside the expectation call is being invoked too soon, so before the database operation to create the item and the page change has had a chance to complete?

I have started wondering if this could be down to the order of promises executed by protractor. I have come across articles talking about control flow in protractor and why there may be occasions when you should hook into the result of a protractor call's promise using .then() - I found this

It got me wondering if I should move the call to my saveBtn.click(), that's called at the end of my page object's createOperator function, into the test suite, so doing something like:

it('should create an operator', function () {
      page.settlementBtn.click();
      page.operatorsBtn.click();
      page.fabBtn.click();
      page.createOperator();
      page.saveBtn.click().then(function(){
        expect(page.headline.getText()).toEqual('Operators');
      });
    });

I'm starting to clutch at straws here to be honest, so any thoughts\advice from the community here would be much appreciated.

Thanks!

Community
  • 1
  • 1
mindparse
  • 6,115
  • 27
  • 90
  • 191
  • 1
    Seems like timing issues. Sometimes Protractor is asking just that little bit too early what the headline text is and it is getting the old page. When transitioning to new pages I tend to use a helper function called something like `waitForUrlToBeLike(x)` which waits for the current url to contain `x`. Stability of Protractor tests I find depends on making sure you wait for things to be how you expect them. – preeve Mar 20 '17 at 11:50
  • @preeve that sounds like something useful I could try, could you share the helper function you currently use? – mindparse Mar 20 '17 at 11:55
  • As a best practice, you may want to reduce the number of actions you are performing in a single `it` block. Why can't you make an assertion after every `click`? Something must be changing with those click events that can be captured, and help with manipulating the control flow. – Gunderson Mar 20 '17 at 12:05
  • @Gunderson - I guess you are referring to the `settlementBtn`, `operatorsBtn` and `fabBtn` clicks in my test snippet I posted?. Those deal with clicking nav menu items and an action button to get me to the create form for filling in input fields, so yes I probably could add some extra assertions here thinking about it. When you say manipulating the control flow, how can assertions affect this? – mindparse Mar 20 '17 at 13:13
  • Because your next click event wont kick off until the previous `it` block has completed. This will help you find the flakey parts of your tests i.e. which actions may need conditional waits added between them – Gunderson Mar 20 '17 at 13:22
  • Ah interesting ok, thanks for the advice! – mindparse Mar 20 '17 at 13:23
  • 1
    @Gunderson - Using preeve's wait for url to change approach and also adding assertions after each interaction as you described has helped me enormously, I haven't had any failures having run many many times after making the changes - thank you so much! – mindparse Mar 21 '17 at 10:30

1 Answers1

2

As requested, here is the function I use for waiting for URLs to be as they should.

public waitForUrlToBeLike (urlPart: string, timeout: number = 10000) {
    return browser.wait(() => {
        return browser.driver.getCurrentUrl().then((url) => {
            let regex = new RegExp(urlPart);
            return regex.test(url);
        });
    }, timeout);
}

I also use the following a lot to wait for elements to be present before making assertions on them:

public waitTillPresent (element: ElementFinder, timeout: number = 10000) {
    return browser.wait(() => {
        return element.isPresent();
    }, timeout);
}
preeve
  • 1,198
  • 12
  • 11
  • Thanks for this, I will give it a go, sounds like it should help! – mindparse Mar 20 '17 at 12:11
  • This has worked a treat for my failing cases, waiting for the url to change after clicking buttons to change states once a http operation has resolved the intermittent failing cases I was seeing on our Jenkins box. Thank you so much for your help! – mindparse Mar 21 '17 at 10:25