Fetching an RSS Feed using Google Apps Script Returns 406 Error

Question

I've got a simple Google apps script that is fetching a series of RSS feeds:

for (let i = 0; i < rssUrls.length; i++) {
   const rssUrl = rssUrls[i];

   const response = UrlFetchApp.fetch(rssUrl);

   console.log(response.getContentText());
}

The URL that's failing is: http://tenfootpole.org/ironspike/?feed=rss2

The response I get is:

<html><head><title>Error 406 - Not Acceptable</title><head><body><h1>Error 406 - Not Acceptable</h1><p>Generally a 406 error is caused because a request has been blocked by Mod Security. If you believe that your request has been blocked by mistake please contact the web site owner.</p></body></html>

It works fine through the browser or POSTman - & even using the =IMPORTFEED() function in google sheets. Not sure what makes GAS unique?

This is the fix I made after reading Google App Script external API return error 406 & Google Apps Script Blocked by Mod Security Error 406 which seem related.

const options = {
  headers: {
    Accept: '*/*'
  },
  'muteHttpExceptions': true
};

const response = UrlFetchApp.fetch(rssUrl, options);

console.log(response.getContentText());

But I still get the same problem. I've tried various combinations of content type, encoding types and languages but I always the same error. I suspect I'm seeing a default response for something else? Any idea what?

Although I'm not sure whether this is included in your expected direction, I proposed a workaround. Please confirm it. If that was not useful, I apologize. — Tanaike, Mar 16 '23 at 01:43

score 0 · Accepted Answer · answered Mar 16 '23 at 01:43

I could correctly replicate your situation. Unfortunately, even when Accept, Accept-Encoding, and Accept-Language are used in the request header, the status code of 406 couldn't be removed. From this situation, I thought that your situation might be related to this thread. In this case, how about the following workaround?

In this workaround, when the response code is 406, the URL is requested using IMPORTFEED or IMPORTDATA on the Spreadsheet. Namely, a Spreadsheet is used as a temporal sheet for retrieving the data from the URL. By this, the values can be obtained.

Sample script:

This script used Spreadsheet. So, please copy and paste the following script to the script editor of Spreadsheet. And, please set the sheet name. This sheet is used as a temporal sheet.

function sample() {
  const sheetName = "Sheet1"; // Please set your sheet name.
  const rssUrls = ["http://tenfootpole.org/ironspike/?feed=rss2", , ,]; // Please set your URLs.

  rssUrls.forEach(url => {
    const response = UrlFetchApp.fetch(url, { muteHttpExceptions: true });
    let res = "";
    const statusCode = response.getResponseCode();
    if (statusCode == 406) {
      const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
      sheet.clear();
      const range = sheet.getRange("A1");
      range.setFormula(`=IMPORTFEED("${url}")`);
      SpreadsheetApp.flush();
      res = sheet.getDataRange().getDisplayValues().join("\n");
    } else if (statusCode == 200) {
      res = response.getContentText();
    } else {
      console.log(`Error occurs at "${url}".`);
      return;
    }

    console.log(res); // Here, you can confirm the retrieved value.

    // do something. Please set your script using the response value.

  });
}

When this script is run, when the response code is 406, the value is retrieved using IMPORTFEED with a temporal sheet.

If you want to use IMPORTDATA instead of IMPORTFEED, please modify it as follows.

From

  range.setFormula(`=IMPORTFEED("${url}")`);
  SpreadsheetApp.flush();
  res = sheet.getDataRange().getDisplayValues().join("\n");

To

  range.setFormula(`=IMPORTDATA("${url}")`);
  SpreadsheetApp.flush();
  res = sheet.getDataRange().getDisplayValues().map(r => r.filter(String).join(",")).join("\n");

References:

By the way, if you want to clear the temporal sheet after the script is run, please add `range.clearContent()` just after the line of `res = sheet.getDataRange().getDisplayValues().join("\n");`. — Tanaike, Mar 16 '23 at 03:56
Thanks - This solution worked - it was what i was considering doing but your example helped a lot! Its certainly less reliable as the IMPORTFEED function can take a little while to load - so i have to put a 1 second sleep in it - but as this is maybe 1 of >300 URLs that get called it not so bad. Thanks! — Olobosk, Mar 20 '23 at 11:19

Fetching an RSS Feed using Google Apps Script Returns 406 Error

1 Answers1

Sample script:

References: