1

I've noticed you can't just launch puppeteer in your script and that it requires you to connected to an existing browserWSEndpoint. I'm trying to scrape data off a website when the user clicks a submit button on a web-page.

I looked at the posts found here but couldn't quite get it to work with any of the solutions. I know you can open up a debugger browser and connect to the endpoint for it, but the link changes every time which is not very convenient especially if it is used by other people.

So far I have this in my node.js server class:

var express = require('express');

var app = express();

app.use('/public', express.static('public'));
app.use('/node_modules', express.static('node_modules')); 

app.get('/', function(req, res){
    res.sendFile(__dirname + '/index.html')
});

app.listen(3000);

I've tried to create a proxy server as suggested in the link but had no luck. My script where I'm trying to run puppeteer looks like this:

const puppeteer = require('puppeteer');

async function getTitle() {
  const browser = await puppeteer.connect({
    browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/9f0a2240-2cb7-4efa-ac3c-8ef883d36d12',
  });
  const page = await browser.newPage();
  await page.goto('http://example.com');
  const title = await page.title();
  await page.close();
  await browser.disconnect();
  return title;
}

getTitle().then(console.log);

The browserWSEndpoint changes every time you reopen the browser as far as I know, which is not ideal since the web app wouldn't work unless you change the source code every time. Also, I'm not even sure how you find the /devtools/browser link. All I found from searching was to type chrome.exe and some flags then navigate to localhost:5200 or something like that. However, I only see /devtools/pages/.. on that page. Though this wasn't the solution I was looking for anyways.

Is there a way to launch a puppeteer instance at some web socket that is created dynamically then have the script connect to that instance of puppeteer when needed? I'm quite new to web sockets, so maybe one of the solutions in the link I have above was what I was looking for, but I didn't quite understand what I was missing.

Gunter
  • 23
  • 6

0 Answers0