I'm learning how to use google apps script to scrape a web page and store the data in a spreadsheet but I am having issues with certain websites where lazy loading is enabled.
function myFunction() {
const webUrl = 'https://www.jumia.com.ng/android-phones/';
const varRespone = UrlFetchApp.fetch(webUrl).getContentText();
const $ = Cheerio.load(varRespone);
$('.info').each(function(index, element){
const name = $(this).find('.name').text();
if(!name){
// Logger.log($(this).html());
}
Logger.log(++index + ' ' + name)
})
}
Take the sample code below which scrapes an eCommerce website and logs the name of each product on the page.
There are 48 products on the page but only forty is logged successfully, the remaining eight comes back blank. I inspected the response from UrlFetchApp.fetch(webUrl).getContentText()
and discovered that lazy loading is enabled for the eight divs coming back blank.
I have tried using the response from UrlFetchApp.fetch(webUrl).getContentText()
to create a template using HtmlService.createTemplate
and then calling .evaluate()
but this is not sucessfull.
function test(){
const webUrl = 'https://www.jumia.com.ng/android-phones/';
const varRespone = UrlFetchApp.fetch(webUrl).getContentText();
const l =HtmlService.createTemplate(varRespone)
let x = l.evaluate().getContent();
const $ = Cheerio.load(x);
$('.info').each(function(index, element){
const name = $(this).find('.name').text();
if(!name){
// Logger.log($(this).html());
}
Logger.log(++index + ' ' + name)
})
}
Is there a way to go around this?