2

In basic example of crawlera & puppeteer, authorization of proxy is done in this way:

await page.setExtraHTTPHeaders({
   'Proxy-Authorization': 'Basic ' + Buffer.from('<api_key>:').toString('base64'),
});

but it gives error: net::ERR_INVALID_ARGUMENT I already know that its due to chromium. In their source code I found this:

// Removing headers can't make the set of pre-existing headers unsafe, but adding headers can.

Therefor, I started using this: await page.authenticate({ username:'<api_key>',password:'' });

and it works perfectly fine with http pages. But with https it gives error: net::ERR_UNEXPECTED_PROXY_AUTH

anyone knows how to workaround that?

My versions:

  • HeadlessChrome/88.0.4298.0
  • Puppeteer: "version": "5.5.0",

Full script:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({
        ignoreHTTPSErrors: true,
        args: [
            '--proxy-server=proxy.crawlera.com:80',
            '--no-sandbox', 
            '--disable-setuid-sandbox'
        ]
    });
    const page = await browser.newPage();
    await page.authenticate({ username:'<apikey>',password:'' });
    
    
    
    try {
        await page.goto('https://httpbin.scrapinghub.com/get', {timeout: 180000});
    } catch(err) {
        console.log(err);
    }
  
    await browser.close();
})();
Tomasz Kurowski
  • 21
  • 1
  • Have you tried this? https://support.scrapinghub.com/support/discussions/topics/22000011294 – Or Assayag Nov 27 '20 at 20:47
  • Yes, just tried. It gives exactly the same error: Error: net::ERR_INVALID_ARGUMENT – Tomasz Kurowski Nov 27 '20 at 20:56
  • 1
    I've managed to do it via proxy-chain. So if anyone has got similar issues, that helped me a lot: https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212 – Tomasz Kurowski Nov 28 '20 at 01:00

0 Answers0