Refresh when an element changes on page

Question

I try to scrap an element on a website and display it on localhost with Puppeteer (1). But when this element changes, I would like to refresh data without opening a new browser/page with Puppeteer and only when element changes (2).

For my example, I use www.timeanddate.com and the element is time (hours and minutes). For moment, only first part works. I don't have solution for second one.

Please find below, my code.

app.js

var app = require('express')();
var server = require('http').createServer(app);
var io = require('socket.io').listen(server);
var puppeteer = require('puppeteer');

app.get('/', function(req, res) { 
    res.render('main.ejs');
});

server.listen(8080);

let scrape = async () => {
    var browser = await puppeteer.launch({headless: true});
    var page = await browser.newPage();
    await page.goto('https://www.timeanddate.com/worldclock/personal.html');
    await page.waitFor(300);
    //await page.click('#mpo > div > div > div > div.modal-body > div.form-submit-row > button.submit.round.modal-privacy__btn');

    var result = await page.evaluate(() => {
        return document.getElementsByClassName('c-city__hrMin')[0].innerText;
    });

    return result;
};

io.sockets.on('connection', function (socket) {
    scrape().then((value) => { // it tooks time, a few seconds while page is loading.
        console.log(value);
        socket.emit('refresh', value);
    });
});

main.ejs

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8" />
        <title>What time is it?</title>
        <style>
            a {text-decoration: none; color: black;}
        </style>
    </head>

    <body>
        <h1>Welcome !</h1>

        <div id="time">loading</div>

        <script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
        <script src="/socket.io/socket.io.js"></script>
        <script>
            var socket = io.connect('http://localhost:8080');

            socket.on('refresh', function (value) {
                $('#time').html(value);
            });         
        </script>
    </body>
</html>

I try Fiverr but awful experience. I hope it will better here :)

Thank you for helping me.

If the answer worked for you, please choose as accepted answer so it doesn't show as unanswered. :) — Md. Abu Taher, Oct 18 '18 at 21:55

Md. Abu Taher · Accepted Answer · 2018-10-19T02:41:45.130

You want to emit event when the data changes. There are multiple ways to do that, such as,

Try to get new data on interval
Look for the change and emit from within browser

Update Code Readability

I will discuss both of them. But first, lets split the code for a better usability. It's completely optional but you should do it.

/**
 * Scraper
 * Use this instead of scrape variable
 */
let browser, page;
const scraper = {
  async open() {
    browser = await puppeteer.launch({ headless: true });
    page = await browser.newPage();
    const url = "https://www.timeanddate.com/worldclock/personal.html";
    await page.goto(url);
    await page.waitFor(300);
  },
  async getTime() {
    return page.evaluate(() => {
      return document.querySelector(".c-city__digitalClock").innerText; // time with seconds 5:43:22am
    });
  }
};

We can add other methods to this object later if we need. This is not the best format, but this will help us understand the code better at this point.

Method 1. Set Interval

Let's modify the connection, we just need to open the page once and poll new data on some interval.

/**
 * Socket Connection Monitor
 */
io.sockets.on("connection", async function(socket) {
  // open the page once
  await scraper.open();

  // start the interval loop
  setInterval(async () => {

    // get the time every second
    const time = await scraper.getTime();

    // emit the updated time
    socket.emit("refresh", time);
  }, 1000); // how many millisecond we want
});

Method 2. Add events to the browser itself.

This is advanced and much more complex, however very accurate.

You can add this inside scraper object.

// <-- Pass the socket so it can use it
async runEvents(socket) {
    // Create a Shadow event tracker on puppeteer
    await page.exposeFunction("emitter", (...data) => {
      socket.emit(...data)
    });
    await page.evaluate(function observeDom() {
      // expose the observer which will watch
      //More Details https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver
      // select the target node
      var target = document.querySelector(".c-city__digitalClock");
      // create an observer instance
      var observer = new MutationObserver(function(mutations) {
        // Do something on change
        emitter("refresh", target.innerText); // <-- trigger the event whenever there is a change
      });

      // configuration of the observer:
      var config = { childList: true, subtree: true };
      // pass in the target node, as well as the observer options
      observer.observe(target, config);
    });
  }

And then your connection will look like,

io.sockets.on("connection", async function(socket) {
  await scraper.open();
  await scraper.runEvents(socket); // <-- Pass the socket
});

How it works,

When the socket is open,
We open the browser and page
We run the events.
- We setup a custom event which will run socket.emit with whatever data it gets
- We expose the custom event on the page.
- We observe the dom element from then,
- Whenever there is a little change, we trigger the custom event we made

Here is a visual difference between these two:

(I used 500ms interval and it's 60 frames per second, so the animation is not catching everything, but it's there, link to repo.)

Difference

The difference between setInterval and the event is, setInterval will check after certain amount of time, while the observer will continuously observe the changes and trigger whenever there is a change.

Which to choose:

If you like simplicity, go with setInterval version.
If you need precision, go with observer version.

Hey Abu, you really rock ! The method #1 is easy to understand and to apply without any problem. **For method #2, I didn't succeed** :(( Can you help a little more to fix it ? The second one seems to be really awesome. — z0om, Oct 19 '18 at 00:42
I updated the code, there was a small typo with passing the socket. — Md. Abu Taher, Oct 19 '18 at 02:08
Really awesome. Thank you a lot ! I replaced "c-city__digitalClock" by "c-city__hrMin" but it works too (initial need). But, when I start, stop and start again app.js, I have this issue : "UnhandledPromiseRejectionWarning: Error: Failed to add page binding with name emitter: window['emitter'] already exists!". Why ? — z0om, Oct 19 '18 at 15:33
I was also wondering that. It is probably getting invoked multiple times for some reason. — Md. Abu Taher, Oct 26 '18 at 11:03
Wow, would like to 100+ this for adding a git repo. That is above and beyond. Kudos. — lharby, Jul 24 '23 at 20:01

Refresh when an element changes on page

1 Answers1

Update Code Readability

Method 1. Set Interval

Method 2. Add events to the browser itself.

Difference

Linked