Unable to parse data from UrlFetchApp in Google Apps Script

Question

My current Cloud Run URL returns a long string, matching the exact format as described here.

When I run the following code in Google Apps Script, I get a Log output of '1'. What happens, is the entire string is put in the [0][0] position of the data array instead of actually being parsed.

function myFunction() {
const token = ScriptApp.getIdentityToken();
const options = {
  headers: {'Authorization': 'Bearer ' + token}
}
var responseString = UrlFetchApp.fetch("https://*myproject*.a.run.app", options).getContentText();
var data = Utilities.parseCsv(responseString, '\t');
Logger.log(data.length);
}

My expected output is a 2D array as described in the aforementioned link, with a logged output length of 18.

I have confirmed the output of my response by:

Logging the responseString
Copying the output log into a separate var -> var temp = "copied-output"
Changing the parseCsv line to -> var data = Utilities.parseCsv(temp, '\t')
Saving and running the new code. This then outputs a successful 2D array with a length of 18.

So why is it, that my current code doesn't work? Happy to try anything because I am out of ideas.

Edit: More information below.

Python script code

@app.route("/")
def hello_world():
    # Navigate to webpage and get page source
    driver.get("https://www.asxlistedcompanies.com/")
    soup = BeautifulSoup(driver.page_source, 'html.parser')

    # ##############################################################################
    #                   Used by Google Apps Script to create Arrays
    # This creates a two-dimensional array of the format [[a, b, c], [d, e, f]]
    # var csvString = "a\tb\tc\nd\te\tf";
    # var data = Utilities.parseCsv(csvString, '\t');
    # ##############################################################################
    long_string = ""
    limit = 1
    for row in soup.select('tr'):
        if limit == 20:
            break
        else:
            tds = [td.a.get_text(strip=True) if td.a else td.get_text(strip=True) for td in row.select('td')]
            count = 0
            for column in tds:
                if count == 4:
                    linetext = column + r"\n"
                    long_string = long_string+linetext
                else:
                    text = column + r"\t"
                    long_string = long_string+text
                    count = count+1
            limit = limit+1
        
    return long_string

GAS Code edited:

function myFunction() {
const token = ScriptApp.getIdentityToken();
const options = {
  headers: {'Authorization': 'Bearer ' + token}
}
var responseString = UrlFetchApp.fetch("https://*myfunction*.a.run.app", options).getContentText();
Logger.log("The responseString: " + responseString);

Logger.log("responseString length: " + responseString.length)

Logger.log("responseString type: " + typeof(responseString))
var data = Utilities.parseCsv(responseString, '\t');
Logger.log(data.length);
}

GAS logs/output as requested:

6:17:11 AM  Notice  Execution started
6:17:22 AM  Info    The responseString: 14D\t1414 Degrees Ltd\tIndustrials\t21,133,400\t0.001\n1ST\t1ST Group Ltd\tHealth Care\t12,738,500\t0.001\n3PL\t3P Learning Ltd\tConsumer Discretionary\t104,613,000\t0.005\n4DS\t4DS Memory Ltd\tInformation Technology\t58,091,300\t0.003\n5GN\t5G Networks Ltd\t\t82,746,600\t0.004\n88E\t88 Energy Ltd\tEnergy\t42,657,800\t0.002\n8CO\t8COMMON Ltd\tInformation Technology\t11,157,900\t0.001\n8IH\t8I Holdings Ltd\tFinancials\t35,814,200\t0.002\n8EC\t8IP Emerging Companies Ltd\t\t3,199,410\t0\n8VI\t8VIC Holdings Ltd\tConsumer Discretionary\t13,073,200\t0.001\n9SP\t9 Spokes International Ltd\tInformation Technology\t21,880,100\t0.001\nACB\tA-Cap Energy Ltd\tEnergy\t7,846,960\t0\nA2B\tA2B Australia Ltd\tIndustrials\t95,140,200\t0.005\nABP\tAbacus Property Group\tReal Estate\t1,679,500,000\t0.082\nABL\tAbilene Oil and Gas Ltd\tEnergy\t397,614\t0\nAEG\tAbsolute Equity Performance Fund Ltd\t\t107,297,000\t0.005\nABT\tAbundant Produce Ltd\tConsumer Staples\t1,355,970\t0\nACS\tAccent Resources NL\tMaterials\t905,001\t0\n
6:17:22 AM  Info    responseString length: 1020
6:17:22 AM  Info    responseString type: string
6:17:22 AM  Info    1.0
6:17:22 AM  Notice  Execution completed

Thanks for the comment. Hopefully the new information i added is helpful. And again, this works perfectly fine if i copy that responseString from the log and paste it directly into the parseCsv. — BlueBell, Jan 09 '21 at 20:22

TheMaster · Accepted Answer · 2021-01-09T21:26:01.510

1

Issue:

Using a r'' raw string flag makes \n and \t, a literal \ and n/t respectively and not a new line or a tab character. This explains why you were able to copy the "displayed" logs to a variable and execute it successfully.

Solution:

Don't use r flag.

Snippet:

    linetext = column + "\n" #no flag
    long_string = long_string+linetext
else:
    text = column + "\t" #no flag

edited Jan 09 '21 at 21:26

answered Jan 09 '21 at 20:57

TheMaster

45,448
6
62
85

Thanks heaps for the response. That makes a lot of sense! However, when I don't use the raw string flag, the '\n' tag literally adds a new line rather than inserting that as text. I also used the raw string flag for '\t' just to "be safe". So how can I insert '\n' into a string like GAS requires – BlueBell Jan 09 '21 at 21:03
@BlueBell Don't use `r` flag. You do need a literal new line and a literal tab character. – TheMaster Jan 09 '21 at 21:23
That is perfect. Works just fine now. Can't believe the answer was so simple though. Thanks! – BlueBell Jan 09 '21 at 21:50

Unable to parse data from UrlFetchApp in Google Apps Script

1 Answers1

Issue:

Solution:

Snippet: