Loading a webpage from a URL as an html to build a price tracker using Firebase Functions

Question

I'm trying to use firebase cloud function to build a price tracker to track from a webpage

let's say I want to track this product: https://www.jarir.com/sa-en/apple-magic-keyboard-mouse-combo-547294.html

How to load this page in my cloud function project using typeScript lang and save it in a const as an html or string ?

Here is i'm using pubsub to trigger this event every 30 minutes:

import * as functions from 'firebase-functions';
import admin = require('firebase-admin');

export const priceTracker = functions.pubsub.schedule('every 30 minutes')
.onRun(async(context) => {

    // 1- How to load this page: https://www.jarir.com/sa-en/apple-magic-keyboard-mouse-combo-547294.html 
    // and save it in a `const` as an html or string 

    // 2- How to extract the price from the html const and save the result into my firestore database
});

Thanks in advance

You need to use a node.js library which allow you to do some web scraping and then analyse the page to extract the desired data. See this blog post, it might help you: https://levelup.gitconnected.com/web-scraping-with-node-js-c93dcf76fe2b. In addition, note that you need to be on the "Blaze" pricing plan. As a matter of fact, the free "Spark" plan "allows outbound network requests only to Google-owned services". See https://firebase.google.com/pricing/ (hover your mouse on the question mark situated after the "Cloud Functions" title) — Renaud Tarnec, Jun 06 '20 at 14:14

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

This is really a broad topic, that is hard to answer completely in a Stack Overflow answer. What I'll do instead is help you break it down into smaller steps, and provide links for those.

In all of these it really helps to realize that:

Cloud Functions are for the most part just small node modules that are managed by Google's machines. So if you want to do something in Cloud Functions, consider how to do it in Node.js.
Node.js is really just JavaScript running on a server. So if you want to do sometihng in Node.js, consider how to do it in "plain old JavaScript".

With that in mind:

Loading a HTML page from a URL into your Cloud Functions code

Searching for Loading a HTML page from a URL in Node.js seems to have some great results, including Get URL Contents in Node.js with Express.
Finding the price in HTML comes down to parsing that HTML. This link seems like a good place to start of that: Extracting table value from an URL with Node JS, with many more good results by searching for Parsing a HTML page from a URL in Node.js
Finally: storing the resulting HTML or value into Firestore should be fairly straightforward by following the documentation on adding data to the database.

Loading a webpage from a URL as an html to build a price tracker using Firebase Functions

1 Answers1