How to capture shadow root element with puppeteer page.screenshot() method?

Question

I am trying to capture screenshot of webpage using puppeteer page.screenshot method. But when I see the output screenshot it misses some part of page which is shadow root element inside html. How can I capture shadow root element also with puppeteer screenshot method. I am currently using below code. Example URL is https://www.buybuybaby.com/

const browser = await puppeteer.launch();

const page = await browser.newPage();

await page.setExtraHTTPHeaders({
      'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
      accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
    })


await page.goto("https://www.buybuybaby.com/", {    //URL with shadow root element
            waitUntil: 'networkidle2',
            timeout: 160000   
})

await page.setViewport({
    width: 1600,
    height: 20000
  })

await page.screenshot({path: 'screenshot.png'});

await browser.close();

Edited my question with whole code and url. – Murtuza Sanjeliwala Sep 19 '21 at 14:22 — Murtuza Sanjeliwala, Sep 19 '21 at 14:22

mapa0402 · Accepted Answer · 2021-10-09T09:32:46.277

0

Page seems to be lazyloading the links/products, try with this awesome answer

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({
        headless: false
    });
    const page = await browser.newPage();
    await page.goto('https://www.buybuybaby.com/', { waitUntil: 'load' });
    await page.setViewport({
        width: 1200,
        height: 800
    });

    const element = await page.$("#trendingProductsList"); 

    await autoScroll(page);

    await page.screenshot({
        path: 'yoursite.png',
        fullPage: true
    });

    await browser.close();
})();

async function autoScroll(page) {
    await page.evaluate(async () => {
        await new Promise((resolve, reject) => {
            var totalHeight = 0;
            var distance = 100;
            var timer = setInterval(() => {
                var scrollHeight = document.body.scrollHeight;
                window.scrollBy(0, distance);
                totalHeight += distance;

                if (totalHeight >= scrollHeight) {
                    clearInterval(timer);
                    resolve();
                }
            }, 100);
        });
    });
}

this is my screenshot

edited Oct 09 '21 at 09:32

answered Sep 20 '21 at 17:07

mapa0402

454
1
8
24

This isn't working. Shadow root elements still missing inside screenshot. – Murtuza Sanjeliwala Oct 03 '21 at 16:39
I tried the code, added { waitUntil: 'load' } and now it seems to work good again :) – mapa0402 Oct 04 '21 at 14:45
Tried same code still same result shadow root elements are missing. – Murtuza Sanjeliwala Oct 06 '21 at 07:43
Can you give an example of what is missing? – mapa0402 Oct 06 '21 at 11:19
I have added output screenshot as you can see it misses footer part. – Murtuza Sanjeliwala Oct 07 '21 at 05:59
I see, updated the code so we get the footer parts. const element = await page.$("#trendingProductsList"); seems to be key for scraping shadow dom element – mapa0402 Oct 07 '21 at 20:13
Still missing. Can you share your output screenshot? – Murtuza Sanjeliwala Oct 08 '21 at 06:41
yes, will post it later today here – mapa0402 Oct 08 '21 at 08:08
May be I got the issue its because of old puppeteer version I am using. – Murtuza Sanjeliwala Oct 08 '21 at 09:01
I am using puppeteer version 5.5.0. Can you help me to solve this for puppteer 5.5.0 version? Thanks. – Murtuza Sanjeliwala Oct 08 '21 at 12:37
Don't sure if 5.5.0 supports the shadow dom/select object by javascript path. I used 10.4.0 for my answer – mapa0402 Oct 09 '21 at 09:34
1

Thanks for all your help. I have resolved the issue with new puppeteer version using your code. – Murtuza Sanjeliwala Oct 09 '21 at 09:50

How to capture shadow root element with puppeteer page.screenshot() method?

1 Answers1