After a few days, I can no longer start Puppeteer until I restart the server

Question

Puppeteer version: 10.2.0
Platform / OS version: Ubuntu 20.04.3 LTS aarch64
Node.js version: v.14.17.6
Happens on any URL.

This is my test script:

bug.hr/homepage.js

const puppeteer = require('puppeteer-extra');

(async () => {
    const eventPage = `https://www.bug.hr/`;
    const browser = await puppeteer.launch({
        headless: true,
        dumpio: true
    });
    const page = await browser.newPage();
    await page.goto(eventPage, {
        waitUntil: 'domcontentloaded'
    });
    console.log("TITLE: ", await page.evaluate(() => document.title));
    await browser.close();
})();

That should start Puppeteer (and it does after I restart the server), instead, I get this:

ubuntu@ip-123-45-67-89:/var/www/tests$ env DEBUG="puppeteer:*" node bug.hr/homepage.js 
  puppeteer:launcher Calling /usr/bin/chromium-browser --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=Translate --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --enable-blink-features=IdleDetection --headless --hide-scrollbars --mute-audio about:blank --remote-debugging-port=0 --user-data-dir=/tmp/puppeteer_dev_chrome_profile-fvYjAc +0ms

DevTools listening on ws://127.0.0.1:42943/devtools/browser/5a94e4ce-f053-4e83-afdb-f44731fc185e
[1109/015336.602361:ERROR:devtools_http_handler.cc(292)] Error writing DevTools active port to file
[1109/015336.608627:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 
[1109/015336.612599:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015336.613361:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015336.613265:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
  puppeteer:protocol:SEND ► {"method":"Target.setDiscoverTargets","params":{"discover":true},"id":1} +0ms
  puppeteer:protocol:RECV ◀ {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"17591FCF231B151133561BADC2D1A20C","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"3F175AC4C3C999720E0FFEFA74971974"}}} +0ms
  puppeteer:protocol:RECV ◀ {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"3066064f-b08d-429f-af6c-99c828566f63","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}} +1ms
  puppeteer:protocol:RECV ◀ {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"7b188175-60bc-4608-b6a2-a83ed34381b0","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}} +0ms
  puppeteer:protocol:RECV ◀ {"id":1,"result":{}} +0ms
  puppeteer:protocol:SEND ► {"method":"Target.createTarget","params":{"url":"about:blank"},"id":2} +4ms
  puppeteer:protocol:RECV ◀ {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"BD017C91F9A3803C4D1204C1F648B61E","type":"page","title":"","url":"","attached":false,"canAccessOpener":false,"browserContextId":"3F175AC4C3C999720E0FFEFA74971974"}}} +4ms
  puppeteer:protocol:RECV ◀ {"id":2,"result":{"targetId":"BD017C91F9A3803C4D1204C1F648B61E"}} +0ms
[1109/015336.626616:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
  puppeteer:protocol:RECV ◀ {"method":"Target.targetCrashed","params":{"targetId":"17591FCF231B151133561BADC2D1A20C","status":"crashed","errorCode":133}} +187ms
[1109/015336.827392:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015336.827421:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 1 time(s)
[1109/015336.834997:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 
[1109/015336.838813:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015336.839148:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
  puppeteer:protocol:RECV ◀ {"method":"Target.targetCrashed","params":{"targetId":"BD017C91F9A3803C4D1204C1F648B61E","status":"crashed","errorCode":133}} +49ms
[1109/015337.039942:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015337.039973:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 2 time(s)
[1109/015337.047216:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 
[1109/015337.051080:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.051332:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.248703:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015337.248732:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 3 time(s)
[1109/015337.255892:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 
[1109/015337.259819:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.260091:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.454161:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015337.454191:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 4 time(s)
[1109/015337.461741:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 
[1109/015337.465935:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.466181:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.666248:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015337.666276:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 5 time(s)
[1109/015337.673524:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 
[1109/015337.677539:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.677807:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.880581:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015337.880610:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 6 time(s)
[1109/015337.882645:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is disabled, ANGLE is 
[1109/015337.884399:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015337.884679:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015338.076122:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015338.076151:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 7 time(s)
[1109/015338.078253:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is disabled, ANGLE is 
[1109/015338.079942:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015338.080236:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015338.268891:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015338.268920:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 8 time(s)
[1109/015338.271026:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is disabled, ANGLE is 
[1109/015338.272679:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015338.272975:FATAL:sandbox_bpf.cc(283)] Kernel refuses to turn on BPF filters
[1109/015338.461377:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=133
[1109/015338.461406:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 9 time(s)
[1109/015338.461419:FATAL:gpu_data_manager_impl_private.cc(417)] GPU process isn't usable. Goodbye.

I can't reproduce it at will. This happens after the server has been running for a week or two weeks. After I restart the server, it works again.

I believe the culprit is Kernel refuses to turn on BPF filters, from the logs.

Source of the error: https://chromium.googlesource.com/chromium/src/+/lkgr/sandbox/linux/seccomp-bpf/sandbox_bpf.cc#282

Can be fixed by passing --disable-seccomp-filter-sandbox argument to Chrome, which introduces security concerns, as it disables Layer-2 of the Chrome sandbox (https://chromium.googlesource.com/chromium/src/+/HEAD/docs/linux/sandboxing.md#the-sandbox-1)

/usr/bin/chromium-browser --product-version gives 95.0.4638.69

It also happens when I run apt update && apt upgrade:

/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524/usr/bin/mandb:
can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter: Unknown error 524
/usr/bin/mandb: can't load seccomp filter/usr/bin/mandb: can't load seccomp filter: Unknown error 524: Unknown error 524

This wasn't happening on Ubuntu-amd64 (now it's Arm).

Any ideas?

Thank you

I would try launching with args: ['--disable-gpu'] and maybe some similar flags. — pguardiario, Nov 14 '21 at 07:11

score 2 · Answer 1 · answered Nov 18 '21 at 10:30

2

Try adding the {args: ['--no-sandbox']} flag in await puppeteer.launch() because it seems to be a sandbox problem given the error FATAL:sandbox_bpf.cc(283)

answered Nov 18 '21 at 10:30

Sid Barrack

1,235
7
17

1

I wouldn’t wanna disable the sandbox. I will test once it stops working again. – The Onin Nov 18 '21 at 19:52
It might also be related to missing chromium dependencies on linux. you could try checking that out before trying to disable the sandbox. – Sid Barrack Nov 19 '21 at 09:37
This did solve it - thanks. Sucks I lose the security of a sandbox, and I guess the best way to make up for it is to wrap each Puppeteer instance within its own VM. – The Onin Nov 20 '21 at 00:13
Still curious to understand the root cause... – The Onin Nov 20 '21 at 00:14
Awesome to hear, like I said I believe its related to chromium dependencies on linux but I'm not really sure it could also be something else depending on your setup. – Sid Barrack Nov 20 '21 at 18:09
How can it be Chromium dependencies if it works until it stops working after a few days? – The Onin Dec 15 '21 at 16:15

The Onin · Answer 2 · 2021-12-16T04:42:44.427

2

Can be fixed by passing --disable-seccomp-filter-sandbox argument to Chromium.

This is bad for security, as it partially disables the sandbox (https://chromium.googlesource.com/chromium/src/+/HEAD/docs/linux/sandboxing.md#the-sandbox-1), but it's the only solution that works for me right now and is better than disabling the whole sandbox or the zygote.

edited Dec 16 '21 at 04:42

answered Dec 15 '21 at 16:14

The Onin

5,068
2
38
55

score 0 · Answer 3 · answered Jan 25 '22 at 08:37

The source of the problem was identified on Ubuntu issue tracker: https://bugs.launchpad.net/ubuntu/+source/libseccomp/+bug/1956954

Specifically, it was identified by Paride Legovini:

Hi Nino, I found a RedHat bug [1] that describes a similar situation (affected is dhclient on ppc64le, but the error is the same).

The suggested workaround is setting

  net.core.bpf_jit_limit = 262144000

via sysctl. I checked and on amd64 I find

  net.core.bpf_jit_limit = 264241152

while on an arm64 machine that's:

  net.core.bpf_jit_limit = 33554432

(both machines are running Bionic), so it looks arch-dependent indeed. Could you please try bumping that value, and check if the issue goes away?

In other words, this wasn't Puppeteer-specific. Rather, it was architecture-specific in Linux kernel.

score -1 · Answer 4 · answered Nov 19 '21 at 04:32

-1

Try to add --single-process and --no-zygote to the args, something like that

const browser = await puppeteer.launch({
    headless: true,
    args: ['--single-process', '--no-zygote']
})

answered Nov 19 '21 at 04:32

Milton Filho

525
3
17

Hey, thank you for your help. I am aware of that argument as well, it’s pretty similar to running Puppeteer without sandbox. In other words, it does sacrifice security. https://codereview.chromium.org/2384163002 – The Onin Nov 19 '21 at 18:51
This crashes with `--no-sandbox should be used together with --no--zygote` pupp exception – The Onin Nov 20 '21 at 00:11

After a few days, I can no longer start Puppeteer until I restart the server

4 Answers4