0

I'm trying to scrape GDP data from BEA website. The following codes returned empty. Can anyone help out what should be changed in the codes? Thanks!

function test() {
  const url = 'https://apps.bea.gov/iTable/iTable.cfm?reqid=19&step=3&isuri=1&nipa_table_list=5&categories=survey';
  var res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
  var $ = Cheerio.load(res);
  var table = $("#tbl_wrapper").find('td').toArray().map(el => $(el).text());
  console.log(table)
}
Newbie
  • 247
  • 3
  • 11
  • Have you tried download the table as CSV instead? Seems like that would be easier. – Granitosaurus Feb 25 '22 at 07:20
  • I know it. But what I wanted is to automate the scraping as I deal with a lot of data mining. Seems like the cheerio can’t handle such a dynamic website that I’m lacking a basic skill. – Newbie Feb 25 '22 at 15:39
  • Cheerio can handle it just fine you just need to invest a bit of reverse engineering skills. However, it's much easier to reverse engineer how the download button works and replicate that in your script instead. – Granitosaurus Feb 26 '22 at 02:42
  • 1
    If the data comes from an xhr you can just load that, otherwise you want a browser like puppeteer. – pguardiario Feb 27 '22 at 09:08
  • Try moving `.toArray()` after `.map()`, so it looks like this `.map().toArray();` – JM-AGMS Mar 11 '22 at 13:57
  • It didn’t help. I have many other applications working well with .toArray().map(). I wonder if you tested it out. – Newbie Mar 12 '22 at 14:21
  • Hi @Tanaike, I learned a lot of web scraping codes from you. Can I ask you again for your help in this post? Thanks! – Newbie Dec 24 '22 at 12:01
  • Does this answer your question? [How can I scrape pages with dynamic content using node.js?](https://stackoverflow.com/questions/28739098/how-can-i-scrape-pages-with-dynamic-content-using-node-js) – ggorlen Jan 04 '23 at 19:15
  • @ggorlen, I’m looking for a solution in Google Apps Script or JavaScript, which isn’t covered in your suggested post. Anyway thank you for a future reference. – Newbie Jan 04 '23 at 19:52
  • You might want to [edit] this post to clarify that. The concept is the same though: you're trying to interact with a dynamic JS-driven site with a static HTML parser that only traverses a plain HTML string. – ggorlen Jan 04 '23 at 20:02

0 Answers0