protractor: random test fail

Question

So I just started work on protractor tests and I'm facing the following problem - my tests fail inconsistently. Sometimes the test may pass and the next time it fails. Reasons to fail is very different, it may because it failed to find an element on a page or element does not have text in it (even if it has).

I'm running on Ubuntu 14.04, the same problem relevant for Chrome Version 71.0.3578.80 and Firefox Version 60.0.2. AngularJS Version 1.7.2 and Protractor Version 5.4.0. I believe the problem is somewhere in my code, so here below I provided an example of an existing code base.

Here is my protractor config

exports.config = {
  rootElement: '[ng-app="myapp"]',
  framework: 'jasmine',
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['./e2e/**/*protractor.js'],
  SELENIUM_PROMISE_MANAGER: false,
  baseUrl: 'https://localhost/',
  allScriptsTimeout: 20000,
  jasmineNodeOpts: {
    defaultTimeoutInterval: 100000,
  },
   capabilities: {
     browserName: 'firefox',
     marionette: true,
     acceptInsecureCerts: true,
     'moz:firefoxOptions': {
       args: ['--headless'],
     },
   },
}

And here capabilities for chrome browser

  capabilities: {
    browserName: 'chrome',
     chromeOptions: {
       args: [ "--headless", "--disable-gpu", "--window-size=1920,1080" ]
     }
  },

And finally, my test kit that failed a few times

const InsurerViewDriver = require('./insurer-view.driver');
const InsurerRefundDriver = require('./insurer-refund.driver');
const { PageDriver } = require('@utils/page');
const { NotificationsDriver } = require('@utils/service');
const moment = require('moment');

describe(InsurerViewDriver.pageUrl, () => {
  beforeAll(async () => {
    await InsurerViewDriver.goToPage();
  });

  it('- should test "Delete" button', async () => {
    await InsurerViewDriver.clickDelete();

    await NotificationsDriver.toBeShown('success');
    await PageDriver.userToBeNavigated('#/setup/insurers');

    await InsurerViewDriver.goToPage();
  });

  describe('Should test Refunds section', () => {
    it('- should test refund list content', async () => {
      expect(await InsurerRefundDriver.getTitle()).toEqual('REFUNDS');

      const refunds = InsurerRefundDriver.getRefunds();
      expect(await refunds.count()).toBe(1);

      const firstRow = refunds.get(0);
      expect(await firstRow.element(by.binding('item.name')).getText()).toEqual('Direct');
      expect(await firstRow.element(by.binding('item.amount')).getText()).toEqual('$ 50.00');
      expect(await firstRow.element(by.binding('item.number')).getText()).toEqual('');
      expect(await firstRow.element(by.binding('item.date')).getText()).toEqual(moment().format('MMMM DD YYYY'));
    });

    it('- should test add refund action', async () => {
      await InsurerRefundDriver.openNewRefundForm();

      const NewRefundFormDriver = InsurerRefundDriver.getNewRefundForm();

      await NewRefundFormDriver.setPayment(`#555555, ${moment().format('MMMM DD YYYY')} (amount: $2,000, rest: $1,500)`);
      await NewRefundFormDriver.setPaymentMethod('Credit Card');

      expect(await NewRefundFormDriver.getAmount()).toEqual('0');
      await NewRefundFormDriver.setAmount(200.05);
      await NewRefundFormDriver.setAuthorization('qwerty');

      await NewRefundFormDriver.submit();
      await NotificationsDriver.toBeShown('success');

      const interactions = InsurerRefundDriver.getRefunds();
      expect(await interactions.count()).toBe(2);

      expect(await InsurerViewDriver.getInsurerTitleValue('Balance:')).toEqual('Balance: $ 2,200.05');
      expect(await InsurerViewDriver.getInsurerTitleValue('Wallet:')).toEqual('Wallet: $ 4,799.95');
    });
  });
});

And here some functions from driver's, that I'm referencing in the test above

  // PageDriver.userToBeNavigated
  this.userToBeNavigated = async function(url) {
    return await browser.wait(
      protractor.ExpectedConditions.urlContains(url),
      5000,
      `Expectation failed - user to be navigated to "${url}"`
    );
  };

  this.pageUrl = '#/insurer/33';

  // InsurerViewDriver.goToPage
  this.goToPage = async () => {
    await browser.get(this.pageUrl);
  };

  // InsurerViewDriver.clickDelete()
  this.clickDelete = async () => {
    await $('[ng-click="$ctrl.removeInsurer()"]').click();
    await DialogDriver.toBeShown('Are you sure you want to remove this entry?');
    await DialogDriver.confirm();
  };

  // NotificationsDriver.toBeShown
  this.toBeShown = async (type, text) => {
    const awaitSeconds = 6;
    return await browser.wait(
      protractor.ExpectedConditions.presenceOf(
        text ? element(by.cssContainingText('.toast-message', text)) : $(`.toast-${type}`)
      ),
      awaitSeconds * 1000,
      `${type} notification should be shown within ${awaitSeconds} sec`
    );
  }

  // InsurerRefundDriver.getRefunds()
  this.getRefunds = () => $('list-refunds-component').all(by.repeater('item in $data'));

// InsurerViewDriver.getInsurerTitleValue
this.getInsurerTitleValue = async (text) => {
    return await element(by.cssContainingText('header-content p', text)).getText();
  };

I can't upload the whole code here to give you better understanding because I have a lot of code till this moment, but the code provided above is the exact sample of approach I'm using everywhere, does anyone see a problem in my code? Thanks.

do u have animatons? if yes - try tests with all them disabled — Petr Averyanov, Dec 16 '18 at 00:35
Did the failure happen on same `it` ? One possible reason I guess, it's you miss `await` in some code line. You can run one test file for many times, if it can can always pass, then run another test file until you find the unstable test file, then run it block in that unstable file to narrow down the unstable it block, use same way to narrow down which code line is not stable, then you get which code line missed await — yong, Dec 17 '18 at 14:30
very good point! though it was my first thought, I double check the code and did not find missing `await`. and it fails, most likely, on different it blocks, inconsistently, so I can't find the cause, driving me crazy — Majesty, Dec 17 '18 at 14:35

score 2 · Answer 1 · answered Dec 18 '18 at 02:17

First of all add this block before exporting your config

process.on("unhandledRejection", ({message}) => {
    console.log("\x1b[36m%s\x1b[0m", `Unhandled rejection: ${message}`);
});

this essentially colorfully logs to the console if you missed async/await anywhere, and it'll give confidence that you didn't miss anything.

Second, I would install "protractor-console" plugin, to make sure there is no errors/rejections in the browser console (i.e. exclude possibility of issues from your app side) and add to your config

plugins: [{
    package: "protractor-console",
    logLevels: [ "severe" ]
}]

Then the next problem that I would expect with these signs is incorrect waiting functions. Ideally you have to test them separately as you develop your e2e project, but since it's all written already I'll tell you how I debugged them. Note, this approach won't probably help you if your actions are less than a sec (i.e. you can't notice them). Otherwise follow this chain.

1) I created run configuration in WebStorm, as described in my comment here (find mine) How to debug angular protractor tests in WebStorm

2) Set a breakpoint in the first line of the test I want to debug

3) Then execute your test line by line, using the created run config.

When you start debugging process, webstorm opens up a panel with three sections: frames, console, variables. When the variables section has a message connected to localhost and no variables listed, this means your step is still being executed. Once loading completed you can see all your variables and you can execute next command. So the main principle here is you click Step Over button and watch for variables section. IF VARIABLES APPEAR BEFORE THE APPS LOADING COMPLETED (the waiting method executed, but the app is still loading, which is wrong) then you need to work on this method. By going this way I identified a lot of gaps in my custom waiting methods.

And finally if this doesn't work, please attach stack trace of your errors and ping me

Sure, if this helped you find what your problem was please accept the answer as correct one. Otherwise lmk what your concerns are. Thanks — Sergey Pleshakov, Dec 18 '18 at 18:19

score 1 · Answer 2 · answered Dec 20 '18 at 00:37

I'm concerned about this code snippet

describe(InsurerViewDriver.pageUrl, () => {
  beforeAll(async () => {
    await InsurerViewDriver.goToPage();
  });

  it('- should test "Delete" button', async () => {
    await InsurerViewDriver.clickDelete();

    await NotificationsDriver.toBeShown('success');
    await PageDriver.userToBeNavigated('#/setup/insurers');

    await InsurerViewDriver.goToPage(); // WHY IS THIS HERE?
  });

  describe('Should test Refunds section', () => {
    it('- should test refund list content', async () => {
      // DOESN'T THIS NEED SOME SETUP?
      expect(await InsurerRefundDriver.getTitle()).toEqual('REFUNDS');
// <truncated>

You should not depend on the first it clause to set up the suite below it. You didn't post the code for InsurerRefundDriver.getTitle() but if that code does not send the browser to the correct URL and then wait for the page to finish loading, it is a problem. You should probably have await InsurerViewDriver.goToPage(); in a beforeEach clause.

score 1 · Accepted Answer · answered Dec 20 '18 at 13:48

After some time research I found what was the problem. The cause was the way I'm navigate through the app.

  this.goToPage = async () => {
    await browser.get(this.pageUrl);
  };

Turns out, that browser.get method is being resolved when url changed, but now when angularjs done compile. I used the same approach in every test kit, that's why my tests were failing inconsistently, sometimes page was not fully loaded before test start.

So here is an approach that did the trick

  this.goToPage = async () => {
    await browser.get(this.pageUrl);
    await browser.wait(EC.presenceOf(`some important element`), 5000, 'Element did not appear after route change');
  };

You should ensure that page done all the compiling job before moving on.

It's true that `browser.get` only waits for the page served by the server to load, after which all of the JavaScript that affects the DOM starts running. However, `element` and `element.all` both wait for Angular to finish updating the DOM before looking for elements (see [how it works](https://www.protractortest.org/#/infrastructure) in detail), so in general you should not have to add an explicit `wait` like this. On the other hand, if you are running AJAX or other JavaScript outside of Angular, then yes, you have to explicitly wait for that to finish. — Old Pro, Dec 21 '18 at 06:32

score 0 · Answer 4 · answered Dec 16 '18 at 16:33

0

It seems this could be due to asynchronous javascript.

browser.ignoreSynchronization = true; has a global effect for all your tests. you may have to set it back to false, so protractor waits for angular to be finished rendering the page. e.g. in or before your second beforeEach function

answered Dec 16 '18 at 16:33

Abdel

582
4
6

I'm using the latest version of protractor, where `ignoreSynchronization` has been deprecated since long ago – Majesty Dec 17 '18 at 09:29

protractor: random test fail

4 Answers4