How to improve puppeteer startup performance during tests

Question

I've written a small crawler with the help of Puppeteer. Now I'm facing the challenge that my tests are rather slowly (> 3 seconds for each test). I've been able to track it down to the launch function of Puppeteer and the usage of Istanbul/nyc.

If I run the test just with mocha, the tests are finished under 400 ms.
But if I additionally use nyc, the duration of the tests exceeds 3000 ms

All that I'm using is

'use strict';
const puppeteer = require('puppeteer');


module.exports = async function startBrowser() {
  const options = {
    args: [
      // '--no-sandbox',
      // '--disable-setuid-sandbox',
      // '--disable-dev-shm-usage',
      // '--disable-accelerated-2d-canvas',
      // '--disable-gpu'
    ],
    headless: false // true
  };

  return await puppeteer.launch(options);
};

Here is the test I'm using:

'use strict';
/* global describe: false, before: false, it: false,
    beforeEach: false, afterEach: false, after: false, window: false, document: false */

const assert = require('assert').strict;
const startBrowser = require('../');
const util = require('util');



describe('Puppeteer', function() {
  let pageManager;

  it('start the browser', async function() {
    this.timeout(10000);

    console.time('startBrowser');
    const browser = await startBrowser();
    console.timeEnd('startBrowser');
    assert(browser);

    console.time('closeBrowser');
    await browser.close();
    console.timeEnd('closeBrowser');
  });

});

I've created a repository with this code and test here. nyc _mocha ./test/*.test.js runs in ~3500ms, mocha ./test/*.test.js takes only 130ms.

What I've tried so far:

different combination of include/exclude nyc flags
updating to latest versions of Puppeteer, nyc and mocha
removing my Puppeteer arguments
searching for Puppeteer & Istanbul related issues (with not much success)
trying headless: true
bypassing all proxies, see this puppeteer issue

What can I do to have tests with coverage be as fast as the tests alone?

Using:

Ubuntu 19.04
node.js 10.15.3

unfortunately the behavior is the same with `headless: true` — seasick, Oct 23 '19 at 13:39
i am not sure but this link could fix take a look, and please check you systems performance [ https://github.com/GoogleChrome/puppeteer/issues/1846 ] — Shubham Tiwari, Oct 23 '19 at 13:48
Thank you for your efforts! I tried the direct proxy server, sadly without success. Systems performance is fine, it is fast if I don't use code coverage. I'm interested if you experience the same problem as I do: There is a repo to reproduce it in under a minute => https://github.com/seasick/puppeteer-slow-nyc — seasick, Oct 23 '19 at 13:58
With mocha your browser is already launched. With puppeteer you are launching a new browser virtually. Maybe if they introduced a state-save feature to it then subsequent tests might run faster but ultimately it's probably going to stay slow. That said, it's still way better than manually clicking through my project every time I (or anyone else) do work. — Jacksonkr, Mar 20 '20 at 18:22

score 4 · Answer 1 · answered Oct 28 '19 at 16:01

I've started to debug Puppeteer and these are my findings:

Puppeteer is unsurprisingly using child_process.spawn() to spawn a new browser
nyc is using spawn-wrap for such child processes
spawn-wrap is reading the whole executable (./node_modules/puppeteer/.local-chromium/linux-686378/chrome-linux/chrome) into memory with fs.readFileSync which is taking an unusually long time to finish

spawn-wraps README delivers some kind of an explanation:

The initial wrap call uses synchronous I/O. Probably you should not be using this script in any production environments anyway. Also, this will slow down child process execution by a lot, since we're adding a few layers of indirection.

For me personally the answer is that I cannot get the same performance for running tests with and without code coverage as long as I use nyc/istanbul.

I've given c8 a shot, and the performance is nearly the same and I can still have code coverage.

Edi Imanto · Answer 2 · 2023-07-22T16:25:40.880

3

Please try this also.

'use strict'
const puppeteer = require('puppeteer')

module.exports = async function startBrowser() {
  const options = {
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-dev-shm-usage',
      '--disable-accelerated-2d-canvas',
      '--no-first-run',
      '--no-zygote',
      '--single-process',
      '--disable-gpu'
    ],
    headless: true
  }

  return await puppeteer.launch(options)
}

edited Jul 22 '23 at 16:25

answered Oct 28 '19 at 08:37

Edi Imanto

2,119
1
11
17

3

unfortunately this does not change the launch performance for me – seasick Oct 28 '19 at 09:14
i've add a screenshot of your test running in my terminal – Edi Imanto Oct 28 '19 at 10:04
and you are saying that with your changes, the time is nearly the same? In your screenshot, the tests are faster, but the command with coverage (`npm run test`) is still taking 6x the time its running without (`npm run test_mocha`) – seasick Oct 28 '19 at 10:08
let me try this as well on my i3 with 8gb of ram ... and let's see the result. :D – gumuruh Sep 02 '20 at 14:47
1

--single-process Is apparently now working in Windows, saving me ~500ms on browser.newPage – Juanjo Martinez Jun 06 '23 at 11:17
Thanks for validating this flag. @JuanjoMartinez – Edi Imanto Jul 22 '23 at 16:25

score 0 · Answer 3 · answered Mar 18 '21 at 13:59

./Chromium --headless --disable-gpu --remote-debugging-port=9222 --devtools=false
browser = await puppeteer.connect({ browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/........xxxxxx..', });
use express to hold it

PS: But ,I can't specify the browserWSEndpoint the url change every time Chromium --headless restart

How to improve puppeteer startup performance during tests

3 Answers3