More information about failed tests?

Question

How can I bubble up more information about a failed test? Ideally I want a second parameter on the except function where I can pass a string that would show up on the console. Something like:

var i = 0;
var j = 0;
expect( i / j, `failed when dividing ${i} ${j}` )

I also tried this:

test.describe("Test description", () => {
    test("Test name", async ({ page }) => {
        ...snip...
        await test.step("Testing DIV0 is fail", async () => {
            await expect(4/0).toBe(5);
        });
       ...snip...
    });
});

The command line might quickly flash the text Testing DIV0 is fail for a 1/1000th of a second. This is not helpful. Basically Playwright is unusable if I can't find the problem I'm trying to discover.

What I see:


  1) myTests.spec.ts:40:2 › Test description 

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: 5
    Received: Infinity

      44 |              await test.step("Testing DIV0 is fail", async () => {
    > 45 |                      await expect(4/0).toBe(5);
         |                                        ^
      46 |              });


        at myTests.spec.ts:45:22
        at TestTypeImpl._step (...\node_modules\@playwright\test\lib\test\testType.js:213:13)
        at Function.step (...\node_modules\@playwright\test\lib\test\transform.js:133:12)
        at ...\tests\myTests.spec.ts:44:14
        at WorkerRunner._runTestWithBeforeHooks (...\node_modules\@playwright\test\lib\test\workerRunner.js:450:7)

  Slow test: basicPageChecks.spec.ts (24s)

  1 failed
    myTests.spec.ts:40:2 › Test description 
  12 skipped

I could just as easily add a comment in the code. Or add logging.

Perhaps this additional data is available in a report?? Am I missing something??

There is not way to pass an extra message in jest expect (and in playwright expects which extend those). In your example the output clearly says ` Expected: 5 Received: Infinity ` so you can see both expected and actual values. What information do you miss in the output? — Yury Semikhatsky, Oct 15 '21 at 18:11
In the first code snippet I gave the example of the level of detail I want: `failed when dividing ${i} ${j}`). I want to be able to pass a string where I can provide what information I would like. Perhaps the code is looping over a dataset and one example fails. It would be nice to get which piece of data failed. Further more I explain that I do not see the string `Testing DIV0 is fail` which would also satisfy my criteria. — user1529413, Oct 15 '21 at 20:11
I was fortunate to speak with someone familiar with playwright it seems that the problem is sort of handled. It's not exactly intuitive. The output from console.log that fall under a `test()` are logged not only at the time of execution but again under a summary that is presented. — user1529413, Oct 21 '21 at 19:33
Just to be clear, the reason that console.log is not an answer is that it _always_ logs. The goal is to log extra information only when necessary. — user1529413, Oct 21 '21 at 20:30
You can always check your condition manually and throw any message that you like: if (i/j !== 5) throw new Erorr(\`failed when dividing ${i} ${j}\`); — Yury Semikhatsky, Oct 22 '21 at 00:36

user1529413 · Accepted Answer · 2021-10-24T03:00:09.467

Borrowing heavily from this answer: Re-throwing exception in NodeJS and not losing stack trace

One good solution (meaning that only errors are logged, it does not require double checking every logical step with an additional if statement, & doesn't triple the number of lines of code with a try/catch wrapping every except) is to do the following:

Add a new class

export class RethrownError extends Error {
    constructor(message, error) {
        super(message);
        this.name = this.constructor.name;
        if (!error) {
            throw new Error("RethrownError requires a message and error");
        } else {
            this.original_error = error;
            this.stack_before_rethrow = this.stack;
            const message_lines =  (this.message.match(/\n/g)||[]).length + 1;
            this.stack = this.stack.split("\n").slice(0, message_lines+1).join("\n") + "\n" + error.stack;
        }
    }
    static expectWithMessage(message, cb) {
        try {
            cb()
        } catch(ex) {
            throw new RethrownError(message, ex);
        }
    }
}

change the except from

await expect(4/0).toBe(5);

to

await RethrownError.expectWithMessage("DIV by zero", () => {expect(4/0).toBe(5); });

Problem with this solution is that the expect line is pretty ugly (line length).

Hopefully someday MS would update expect() to accept a second error message parameter.

More information about failed tests?

1 Answers1