Node.js puppeteer - Downloading and opening a .idx file

Question

I'm using node.js and puppeteer to get some data. I can click/download a .idx file ... but how can I then open it and process the data?

const tableRows = await page.$$('table > tbody tr');
console.log(tableRows.length);
let tableCell01;
let tableCell01Val;

for (let i=1; i < tableRows.length; i++){
  tableRow = tableRows[i];
  tableCell01 = await tableRow.$('td:nth-child(1) a');
  tableCell01Val = await page.evaluate( tableCell01 => tableCell01.href, tableCell01 );

  const tableLink = await page.$('table > tbody tr td:nth-child(1) a');
  const tableLinkVal = await page.evaluate( tableLink => tableLink.href, tableLink );

  console.log(tableLinkVal);
  await page.goto(tableLinkVal, {waitUntil: 'load'});
}

I can see that it downloads ... but then I get this error

Error: net::ERR_ABORTED at /file.20180702.idx
at navigate (/node_modules/puppeteer/lib/Page.js:602:37)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)

Cody Ｇ · Answer 1 · 2018-09-03T15:12:40.673

0

Downloading files is not supported... yet.

https://github.com/GoogleChrome/puppeteer/issues/299

You could use other file download methods available to node.js, in order to test your download.

Built in Node.js:

How to download a file with Node.js (without using third-party libraries)?

Modules: Axios, Request

Downloading images with node.js

edited Sep 03 '18 at 15:12

answered Aug 31 '18 at 17:36

Cody Ｇ

8,368
2
35
50

What other methods could I use and how would they look like in my case? – Philipp M Sep 03 '18 at 07:22
I added some links. That's something that I could answer but its already been answered before. Hope the links help :) – Cody Ｇ Sep 03 '18 at 15:13

Keith John Hutchison · Answer 2 · 2019-02-09T09:47:55.727

Handling the error worked for me.

    /*
        https://stackoverflow.com/questions/46919013/puppeteer-wait-n-seconds-before-continuing-next-line#46965281
        https://duckduckgo.com/?q=puppeteer+await+3+seconds&atb=v110-5_b&ia=qa
    */
    function delay(time) {
        return new Promise(function(resolve) {
            setTimeout(resolve, time)
        })
    }

    await page.goto(href).catch(function(err){
        console.log('Ignore error')
    }) 

    await delay(1000)
    var tried = 0
    var bail = 10
    var downloaded = fs.existsSync(filename)
    while ( downloaded==false && tried < bail ) {
        tried++
        await delay(1000)
        console.log("fs.existsSync('" + filename +  "')")
        downloaded = fs.existsSync(filename)
    }
    if ( downloaded ) {
        console.log('Downloaded: ' + href + ' filename: ' + filename) ; 
        rename_file(filename,'../sabai/questions_tags.csv')
    } else {
        console.log('Not downloaded: ' + href + ' filename: ' + filename) ;             
    }


example log extract

Downloading: http://localhost:8000/wp-admin/admin.php?page=sabai/questions/tags&file=questions_tags-20190209.csv&q=%2Fquestions%2Ftags%2Fexport%2Fdownload filename: ../sabai/questions_tags-20190209.csv
Downloaded: http://localhost:8000/wp-admin/admin.php?page=sabai/questions/tags&file=questions_tags-20190209.csv&q=%2Fquestions%2Ftags%2Fexport%2Fdownload filename: ../sabai/questions_tags-20190209.csv


I tried using the built-in method for node, axios and requests as suggested in [Downloading files is not supported... yet.][1] but the file was never downloaded.


  [1]: https://stackoverflow.com/a/52120359/162358

Node.js puppeteer - Downloading and opening a .idx file

2 Answers2