3

I would like to load random list of referers from my default location path , for example: 'referers.txt' instead of adding direct "facebook url as referer.

My code:

browser = await puppeteer.getBrowserInstance(port);
const page = await browser.newPage();
page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);
page.on('error', handlePageCrash(page));
page.on('pageerror', handlePageCrash(page));
page.setExtraHTTPHeaders({ referer: 'https://www.facebook.com/' });
theDavidBarton
  • 7,643
  • 4
  • 24
  • 51
Tran
  • 159
  • 1
  • 4
  • 8

1 Answers1

5

Instead of a txt you should choose JSON to store your list of referer values in an array.

referers.json

["https://www.google.com", "https://www.facebook.com", "https://www.instagram.com"]

Then you will be able to pick a random element form the array by: array[randomIndex]. To generate a random number for the length of your array you have multiple possibilities, Math.floor(Math.random() * array.length) only one of them.

referers.js

const puppeteer = require('puppeteer')
const referers = require('./referers.json')

async function fn() {
  const randomReferer = referers[Math.floor(Math.random() * referers.length)]
  console.log(referers)
  console.log(randomReferer)
  const browser = await puppeteer.launch({ headless: false, devtools: true })
  const page = await browser.newPage()

  page.setExtraHTTPHeaders({ referer: randomReferer })

  await page.goto('https://www.instagram.com/')
}
fn()

output example:

[
  'https://www.google.com',
  'https://www.facebook.com',
  'https://www.instagram.com'
]
https://www.facebook.com
theDavidBarton
  • 7,643
  • 4
  • 24
  • 51
  • Can I do the same way for Useragents? – Tran Sep 19 '20 at 18:35
  • sure, why not? actually the solution I gave is rather Node/JavaScript answer to your problem. You can randomize anything like this, then re-use the strings as values for puppeteer methods. – theDavidBarton Sep 19 '20 at 20:28
  • with random useragents I get error I will create new question if you want to check later thx – Tran Sep 19 '20 at 20:34
  • Ok I made new question here pls: https://stackoverflow.com/questions/63973252/puppeteer-browser-useragent-list – Tran Sep 19 '20 at 20:45
  • It shows a warning Provisional headers warning in the network tab – SkyRar Jul 19 '23 at 19:42
  • @SkyRar it may answers your problem: https://stackoverflow.com/questions/21177387/caution-provisional-headers-are-shown-in-chrome-debugger. if those don't help, it would be useful to know which version of puppeteer with which chromium you use it – theDavidBarton Jul 20 '23 at 07:17