0

I'm trying to fetch the top 20 trends from the following page: https://trends.google.com/trends/trendingsearches/daily?geo=US using UrlFetchApp and Cheerio but I keep getting the following content for my divs:

You are using unsupported browser. Some features may not work correctly. Upgrade to a modern browser, such as Google Chrome.Trends has upgraded to a newer version, which is not supported by this device.dismiss, You are using unsupported browser. Some features may not work correctly. Upgrade to a modern browser, such as Google Chrome.Trends has upgraded to a newer version, which is not supported by this device.dismiss, , Google apps, Google apps

I've tried setting the header option to a bunch of different combinations but all rendered uselessly. I've also tried using Postman and it came out with the same result.

This is my base code:

function getSearchTrends(url) {
  var response = UrlFetchApp.fetch(url);
  var status_code = response.getResponseCode();
  if (status_code == 200) {
    var content = response.getContentText();
    const $ = Cheerio.load(content)
    const divs = [];
    
    $('div').each(function(i, elem) {
      divs[i] = $(this).text();
    });
    Logger.log(divs);
  }
}

Is there some sort of workaround to get a useful HTML response? Thanks.

Rubén
  • 34,714
  • 9
  • 70
  • 166
naderabdalghani
  • 1,139
  • 2
  • 14
  • 21
  • https://stackoverflow.com/questions/50124981/using-apps-script-to-scrape-javascript-rendered-web-page – Cooper Nov 01 '20 at 19:10
  • Can I ask you about the concrete output values you expect? Unfortunately, I cannot understand bout the detail of it from `the top 20 trends`. I apologize for this. – Tanaike Nov 01 '20 at 23:24
  • @Tanaike Think nothing of it! If you go to the link mentioned, you would see the top 20 trending search topics in the US. Those values can be parsed from the received HTML content of this page. What I am expecting from UrlFetchApp is the same HTML content received by a browser but I get the HTML content mentioned above. – naderabdalghani Nov 01 '20 at 23:47
  • 2
    That data is coming from a [xhr](https://trends.google.com/trends/api/dailytrends?hl=en-US&tz=-480&geo=US&ns=15). Cheerio won't load it. – pguardiario Nov 02 '20 at 02:02
  • I'm not using Cheerio to load the data. I use Cheerio to parse the received HTML content. – naderabdalghani Nov 02 '20 at 02:05
  • 1
    It won't be there, is what I'm saying. It's coming from the xhr, not the main response. – pguardiario Nov 02 '20 at 02:39
  • Oh, I can now see what the problem is. Anyways, I've managed to get over this problem by using Python and Google Sheets API. This option offers much more than Google Apps Scripts. – naderabdalghani Nov 03 '20 at 04:12

0 Answers0