1

I am trying to use the web scraper from https://github.com/BlakeStevenson/populartimes.js in my React JS project for school.

I keep getting an error that says: ../node_modules/populartimes.js/node_modules/puppeteer/lib/WebSocketTransport.js Module not found: Can't resolve 'ws' in ...

I did some research and I believe its an issue with how puppeteer interacts with React JS. Is there anyway I can fix this?

This is all the code that I am using that is associated with the scraper:

import React, { Component } from 'react';
import { Map, GoogleApiWrapper, InfoWindow, Marker } from 'google-maps-react';

let temp;
const populartimes = require('populartimes.js');
populartimes('ChIJKznozuF0hlQRbfbe7fJJ1rM').then(out => {temp=out.now.percent});

I have installed both websocket and puppeteer.

ZKJ
  • 167
  • 4
  • 15

1 Answers1

2

The problem is that you are trying to use populartimes.js/puppeteer in your React.Js app (on the client/frontend side) which is not possible at the moment. Unpackaged versions of puppeteer neither won't work even if you've added them in your HTML source.

You need to use populartimes.js/puppeteer on the server (backend) side. So if your app is React.Js-only so far: you will need a Node.Js server as well to run the scrapers and to make their output data available to your React.Js app via an endpoint (most probably you would choose Express for this purpose).

Edit: To get some ideas about the relationship between the Node.Js backend, Express and React.Js client app I can recommend this tutorial on dev.to.

theDavidBarton
  • 7,643
  • 4
  • 24
  • 51
  • Thank you! Are there any tutorials you would recommend for learning how to do that? – ZKJ Jun 17 '20 at 19:29
  • There are too many articles on dev.to, Medium and freecodecamp if you search for "React+Express app". Many of them are overkill for your current project and many of them are written in a poor quality. I can recommend this one: https://dev.to/nburgess/creating-a-react-app-with-react-router-and-an-express-backend-33l3 to get an idea how to connect your frontend to the backend. You need to figure out how to combine the scraper module with the index.js (or server.js), but you need to make sure they are on the same level, so not in the client app ;) I extend my answer with this tutorial link. – theDavidBarton Jun 17 '20 at 19:44
  • I'll give it a shot, thank you so much! – ZKJ Jun 17 '20 at 19:46
  • You will see your project grows twice of its size in no time . But it worths: most business logic can be implemented only on the backend side. And with Node.Js: everything is possible ;) – theDavidBarton Jun 17 '20 at 19:50