46

I had to request the same webpage twice to get the cookies in the 1st request and use it in the 2nd request in the following example.

Could anybody show me the code to save the cookies in one puppeteer session and load it in another session so that there is no need to request the same webpage twice in the 2nd session? Thanks.

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://www.genecards.org/cgi-bin/carddisp.pl?gene=BSCL2');
    await page.goto('https://www.genecards.org/cgi-bin/carddisp.pl?gene=BSCL2');

    const linkHandlers = await page.$x("//div[@id='enhancers']//a[@data-track-event='Table See-All']");

    if (linkHandlers.length > 0) {
        const [response] = await Promise.all([
            page.waitForResponse(response => response.url().includes('/gene/api/data/Enhancers')),
            linkHandlers[0].click()
        ]);
        const resp_text = await response.text();
        console.log(resp_text);
    } else {
        throw new Error("Link not found");
    }
    await browser.close();
})();
GabLeRoux
  • 16,715
  • 16
  • 63
  • 81
user1424739
  • 11,937
  • 17
  • 63
  • 152
  • An easier (more reliable) method may be to reuse the entire session: https://stackoverflow.com/questions/57987585/puppeteer-how-to-store-a-session-including-cookies-page-state-local-storage?noredirect=1&lq=1 – FrozenKiwi Jan 26 '23 at 12:35

2 Answers2

72

To save the cookies, you can use the function page.cookies. To reuse the cookies, you can use the page.setCookies function.

Save cookies to disk

const fs = require('fs').promises;

// ... puppeteer code
const cookies = await page.cookies();
await fs.writeFile('./cookies.json', JSON.stringify(cookies, null, 2));

This will read the cookies for the current URL and save them to disk via JSON.stringify and fs.writeFile.

Reuse cookies

const fs = require('fs').promises;

// ... puppeteer code
const cookiesString = await fs.readFile('./cookies.json');
const cookies = JSON.parse(cookiesString);
await page.setCookie(...cookies);

To reuse the cookies, read the files from the disk via fs.readFile. Then parse the file content via JSON.parse. After that you have to call the page.setCookie function. As the function expects the cookies as arguments (and not as one array argument with cookies), we rely on the spread operator, which allows to call the setCookie function with the given cookies array as individual arguments.

Thomas Dondorf
  • 23,416
  • 6
  • 84
  • 105
  • Thanks but unfortunately this doesn't seem to work with the latest version of Puppeteer. The `fs.writeFileSync` results in: ```Error: TypeError: Cannot read property 'writeFileSync' of undefined``` Would you have any suggestion how to fix this? – RocketNuts Sep 18 '19 at 20:48
  • Actually I have created a [minimal example.js](https://pastebin.com/raw/bX438W1q) that does just this. If I run that with `node` I'm getting that same error. – RocketNuts Sep 18 '19 at 20:49
  • @RocketNuts Thanks for the hint, I made a mistake there by mixing the old fs-API with the new promises API. The code is now updated. – Thomas Dondorf Sep 19 '19 at 05:39
  • Thanks, while experimenting I also noticed it seemed to work when I just did `const fs = require('fs');` at the top (without the `.promises`). I actually don't even know what that means. Is there a reason to prefer `writeFile` over `writeFileSync` or vice versa? – RocketNuts Sep 19 '19 at 09:11
  • @RocketNuts The `.promises` is basically the new part of the API. If possible, you should use `writeFile` instead of `writeFileSync` as it will not block the event loop. – Thomas Dondorf Sep 19 '19 at 17:53
  • Thomas, thanks, I'm rather unexperienced with promises, will look into that. What do you mean by blocking the event loop, isn't that something that only applies when using all this in a website, and not (like me) in a single stand-alone NodeJS script that just runs locally? I actually *need* `writeFile` (or `writeFileSync`) to wait i.e. stall the script's further execution until writing the file is finished. – RocketNuts Sep 29 '19 at 12:53
  • 1
    This does not save nor restore HttpOnly cookies, so it is not a complete solution to keep session etc. – user239558 Aug 12 '20 at 18:52
  • No matter what fs API I'm using this is not working – Sebastian Nov 20 '20 at 20:59
  • 1
    This example totally works, confirm. It saves httpOnly cookies. Thanks. – Shinebayar G Jan 07 '22 at 02:50
3

You can follow this code example below:

const puppeteer = require('puppeteer');
const fs = require('fs').promises; //for working with files

//save cookie function
const saveCookie = async (page) => {
    const cookies = await page.cookies();
    const cookieJson = JSON.stringify(cookies, null, 2);
    await fs.writeFile('cookies.json', cookieJson);
}

//load cookie function
const loadCookie = async (page) => {
    const cookieJson = await fs.readFile('cookies.json');
    const cookies = JSON.parse(cookieJson);
    await page.setCookie(...cookies);
}

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await loadCookie(page); //load cookie
  await page.goto('https://example.com');
  await saveCookie(page); //save cookie
  await browser.close();
})();
Sajid Mahamud
  • 39
  • 1