1

I want to:

  1. download a large zip file (curl),
  2. if #1 is successful, unzip the file to a target directory (unzip)
  3. if #2 is successful, delete the file (rm)

I am writing a simple nodejs script to do the above, and am using the child_process.execSync for this.

const execSync = require('child_process').execSync
execSync(`curl --output ${source} ${target}`)
execSync(`unzip -q ${target} -d ${target-dir}`)
execSync(`rm ${target}`)

I am using the *sync version of exec to ensure things happen in sequence. Since these are long-ish running processes (large zip archive with lots of files inside), I would like to see a progress as time goes on. I can either use something like npm progress or, since curl and unzip already show progress, I wouldn't mind using just that. How do I go about achieving this?

Update: here is what I've tried so far. I can get a progress bar for the download, but I am stuck with staring at a blank screen with unzip (which does take a long time because the archive is really large). I could simply perform unzip without the quiet option q but then I would get a listing of every file unzipped. I don't want that. I simply want a progress bar. I tried using node modules unzip and others like that, but they don't work.

const spawn = require('child_process').spawn
const ProgressBar = require('progress')
const http = require('http')
const fs = require('fs')

const target = fs.createWriteStream(target_file)
const req = http.request({ hostname: hostname, path: filename })

req.on('response', function(res) {
    const len = parseInt(res.headers['content-length'], 10)
    
    const bar = new ProgressBar(`downloading ${hostname}/${filename} [:bar] :rate/bps :percent :etas`, {
        complete: '=',
        incomplete: ' ',
        width: 20,
        total: len
    })

    res.on('data', function (chunk) {
        bar.tick(chunk.length);
        target.write(chunk);
    })

    res.on('end', function () {
        target.end(function () {
            console.log(`downloaded ${len} bytes to data/${filename}`)
            
            const unzip = spawn('unzip', ['-q', filename, '-d', target_dir])
            
            unzip.on('close', (code) => {
                console.log(`unzip exited with code ${code}`)

                const rm = spawn('rm', [new_filename])
                rm.on('close', (code) => {
                    console.log(`rm exited with code ${code}`)
                })
            })
        })
        
    })
}).on('error', (err) => {
    console.log('Error: ', err.message)
})

req.end()
punkish
  • 13,598
  • 26
  • 66
  • 101
  • Using async can still ensure sequentiality and keeps the process from hanging waiting for responses. Beyond that, why not make a stab at integrating the progress bar and ask a concrete question about your attempt if you get stuck? This feels like a "implement it for me" question as written. – ggorlen Aug 12 '20 at 18:31
  • first, since my program *has* to work sequentially (I can unzip a file only *after* it has downloaded, and delete it only *after* it has been unzipped), trying to shoehorn `async` operations into a sequence doesn't accomplish anything. But my bigger problem is catching the progress from the shell commands. `exec` buffers the output and spits it out when the command exits. I can use `spawn` to listen to events but from what I understand, shell commands don't really have events that can be used to show a progress. I've added some code I've tried to the OP… part of it works, part of it doesn't – punkish Aug 13 '20 at 05:36
  • No, the issues are related. If you do something sync, you're clogging the progress and the JS engine can't do other things like check the progress and update a status bar. `await` is the same as `then` so you're chaining promises dependently one after the other. Callbacks also work. Sync is basically 99.999% of the time the wrong way to go about dealing with anything IO-bound as is the situation here. Node also has builtin funcs that request resources and manipulate files. See [this](https://stackoverflow.com/questions/18323152/get-download-progress-in-node-js-with-request), for example. – ggorlen Aug 13 '20 at 05:43
  • Your update looks better, aside from the pyramid of callback doom, but that's beside the point. https://www.npmjs.com/package/progress-extract or https://askubuntu.com/questions/909918/q-how-to-show-unzip-progress might help. – ggorlen Aug 13 '20 at 05:48

0 Answers0