17

We use jest to test our API and have quite complex scenarios. We use the beforeAll functions to set up general helper variables for each test and sometimes to set up tenant separation, in other cases we use the beforeEach functions to set up tenant separation for the tests, with some default configuration for the test tenant, ...

For instance a test could like something like this (as you can see, we use TypeScript to write the tests, in case that would matter):

let apiClient: ApiClient;
let tenantId: string;

beforeAll(async () => {
    apiClient = await getClientWithCredentials();
});

beforeEach(async () => {
    tenantId = await createNewTestTenant();
});

describe('describing complex test scenario', () => {
    it('should have some initial state', async () => {
        await checkState(tenantId);
    });

    it('should have some state after performing op1', async () =>{
        await op1(tenantId);
        await checkStateAfterOp1(tenantId);
    });

    it('should have some state after performing op2', async () =>{
        await op2(tenantId);
        await checkStateAfterOp2(tenantId);
    });

    it('should have some state after performing op1 and op2', async () =>{
        await op1(tenantId);
        await op2(tenantId);
        await checkStateAfterOp1AndOp2(tenantId);
    });

    it('the order of op1 and op2 should not matter', async () =>{
        await op2(tenantId);
        await op1(tenantId);
        await checkStateAfterOp1AndOp2(tenantId);
    });    
});

describe('another similar complex scenario', () => {
    // ... you see where this is going
});

The problem is, what's the best way to share those variables initialized by beforeAll and beforeEach? - the above test works if executed with the --runInBand option, which "... runs all tests serially in the current process ..."

But it starts failing pretty randomly, when executed in parallel, mostly referring to tenantId being undefined. Given that those tests are part of ~200 similar tests, serially all pass. In parallel it depends on the machine. The build agent which has 8 cores / 16 threads has only 50-60% passing tests. My colleagues with quad core CPUs have ~80% passing tests and for me with dual core CPU sometimes only 1-2 tests fail, other times ~10. So obviously it depends on the amount of parallelism.

I've found 2 GitHub issues where people were mentioning the possibility to use this to share the context (which doesn't work anymore) or to encapsulate everything in describe:

So I tried a very naive approach:

describe('tests', () => {
    let apiClient: ApiClient;
    let tenantId: string;

    beforeAll(async () => {
        apiClient = await getClientWithCredentials();
    });

    beforeEach(async () => {
        tenantId = await createNewTestTenant();
    });

    describe('describing complex test scenario', () => {
        it('should have some initial state', async () => {
            await checkState(tenantId);
        });

        it('should have some state after performing op1', async () =>{
            await op1(tenantId);
            await checkStateAfterOp1(tenantId);
        });

        it('should have some state after performing op2', async () =>{
            await op2(tenantId);
            await checkStateAfterOp2(tenantId);
        });

        it('should have some state after performing op1 and op2', async () =>{
            await op1(tenantId);
            await op2(tenantId);
            await checkStateAfterOp1AndOp2(tenantId);
        });

        it('the order of op1 and op2 should not matter', async () =>{
            await op2(tenantId);
            await op1(tenantId);
            await checkStateAfterOp1AndOp2(tenantId);
        });    
    });

    describe('another similar complex scenario', () => {
        // ... you see where this is going
    });
});

But this didn't seem to have any effect. I'd really like to run the test in parallel, but I couldn't find anything regarding that in the documentation. Maybe I don't know what I should be looking for?

peter
  • 14,348
  • 9
  • 62
  • 96
  • 1
    I just stumbled upon the exact same problem/question? Did you figured it out in the meantime? – Vetterjack Feb 13 '19 at 19:36
  • 3
    @Vetterjack I don't know what changed, maybe a new version of jest came out or we had a bug somewhere else, but the method described in my question seems to work for us now. We are running tests with 32 workers in parallel for the last 4 months and didn't experience any weird issues where the only explanation would be that the data got mixed up, so I guess it's working – peter Feb 14 '19 at 12:02
  • Thx! I will try it out. Btw. which jest version are you using at the moment? – Vetterjack Feb 15 '19 at 10:12
  • @Vetterjack `23.6.0` – peter Feb 17 '19 at 14:42
  • 1
    I am not clear what you mean by sharing? isn't the issue you are having based on the fact that tenandId is indeed being shared by all tests possibly running at same time? Wouldn't you just want to initialize `const tenantId = await createNewTestTenant();` inside of each tests? – cyberwombat Jan 04 '20 at 02:18

1 Answers1

3

would this work for you?

describe('tests', () => {
    let apiClient: ApiClient;
    let tenantIds: {id: string, used: boolean}[];

    const findUnusedTenantId = () => {
      const tenant = tenantIds.find(a => !a.used);
      tenant.used = true; 
      return tenant.id
    }

    beforeAll(async () => {
        apiClient = await getClientWithCredentials();
    });

    beforeEach(async () => {
        const id = await createNewTestTenant();
        tenantIds.push({id, used: false})
    });

    describe('describing complex test scenario', () => {
        let tenantId: string
        it('should have some initial state', async () => {
            tenantId = fineUnusedTenantId();
            await checkState(tenantId);
        });

        it('should have some state after performing op1', async () =>{
            await op1(tenantId);
            await checkStateAfterOp1(tenantId);
        });

        // ...
    });
    describe('next scenario', () => {
        let tenantId: string
        it('first test', async () => {
            tenantId = fineUnusedTenantId();
            await checkState(tenantId);
        });

you'd prob want to have an afterAll hook to clean up the db

lonewarrior556
  • 3,917
  • 2
  • 26
  • 55