0

I am trying to write to disk multiple images concurrently in NodeJS, wait for them all to be done and then continue with the execution of my program. Here is my code :

const Router = require('express-promise-router');
const fs = require('fs');
const path = require('path');
const multer = require('multer');
const router = new Router();
const axios = require("axios");
var Promise = require('bluebird')
// export our router to be mounted by the parent application
module.exports = router;

async function downloadImage(folder, url, username) {
    const filePath = path.resolve(folder, username + '.jpg');
    const writer = fs.createWriteStream(filePath);

    const response = await axios({
        url,
        method: 'GET',
        responseType: 'stream'
    });

    response.data.pipe(writer);

    return new Promise((resolve, reject) => {
        writer.on('end', resolve(username));
        writer.on('error', reject("error"));
    });
}

router.post('/download_photos', (req, res, next) => {

    let input = [
        {username: "a_S", "image_path": "http://sr.photos2.fotosearch.com/bthumb/CSP/CSP992/k14639982.jpg"},
        {username: "b_L1", "image_path": "https://thumbs.dreamstime.com/z/example-stamp-28420393.jpg"},
        {username: "c_L2", "image_path": "https://thumbs.dreamstime.com/z/example-red-stamp-text-white-43363006.jpg"},
        {username: "d_S", "image_path": "http://sr.photos2.fotosearch.com/bthumb/CSP/CSP992/k14639982.jpg"},
        {username: "e_L2", "image_path": "https://thumbs.dreamstime.com/z/example-red-stamp-text-white-43363006.jpg"},
        {username: "f_L2", "image_path": "https://thumbs.dreamstime.com/z/example-red-stamp-text-white-43363006.jpg"},
        {username: "g_S", "image_path": "http://sr.photos2.fotosearch.com/bthumb/CSP/CSP992/k14639982.jpg"},
        {username: "h_L3", "image_path": "https://www.goldsteinlegal.co.uk/wp-content/uploads/2017/04/Brand-stock-image.jpg"},
        {username: "i_L3", "image_path": "https://www.goldsteinlegal.co.uk/wp-content/uploads/2017/04/Brand-stock-image.jpg"} ,
        {username: "j_L3", "image_path": "https://www.goldsteinlegal.co.uk/wp-content/uploads/2017/04/Brand-stock-image.jpg"}                   ,
    ];

    let folderPath = path.resolve(process.cwd(), "images", "test");
    fs.rmdirSync(folderPath, {recursive: true})
    fs.mkdirSync(folderPath);

    Promise.map(input, (x) => {
            let image_path = x.image_path;
            if (image_path !== "") {
                return downloadImage(folderPath, x.image_path, x.username)
                    .then((res) => x.username)
                    .catch((err) => err);
            }
        },
        {concurrency: 100})
        .then((images) => {
            res.send("OK");
        })
        .catch((err) => res.status(400).send(err));
});

I would have expected the "OK" to be sent only once all the downloading was done. But in fact the server sends back "OK" but the downloading is still happening in the background. (It might not look like it when you execute the code on a fast connection, but try it with a lot more images or much larger ones and the issue should become obvious).

How do I modify that code so that it gets to the res.send("OK") line only when all images are actually done downloading ?

Chapo
  • 2,563
  • 3
  • 30
  • 60
  • `writer.on('end', resolve(username));` does not attach an event handler, but immediately calls the `resolve` function – Bergi Feb 08 '21 at 10:25
  • Thanks but I'm a newbie (as you might have noticed :) ). How should I modify the code so that it does what I've described ? – Chapo Feb 08 '21 at 12:10
  • `await new Promise((resolve, reject) => { writer.on('end', resolve); writer.on('error', reject); }); return username;` – Bergi Feb 08 '21 at 12:11
  • See also https://stackoverflow.com/q/40707812/1048572, https://stackoverflow.com/q/37365268/1048572 or https://stackoverflow.com/a/43084615/1048572 – Bergi Feb 08 '21 at 12:15
  • thanks for the links will have a look. Using your modification, I can see that it downloads everythig but it does not ever continue the execution then i.e never reaches the `res.send("OK")` – Chapo Feb 08 '21 at 12:29
  • Do the individual promises (that are returned by `downloadImage`) fulfill? – Bergi Feb 08 '21 at 12:36
  • I'm not sure how to test that tbh. A breakpoint within the `then` of `return downloadImage(folderPath, x.image_path, x.username).then((res) => x.username).catch((err) => err);` never gets activated. – Chapo Feb 08 '21 at 13:12
  • and the `return username;` within `downloadImage` is never reached either. – Chapo Feb 08 '21 at 13:25
  • AH! the writable stream event should be `finish` and not `end` in the async function ! – Chapo Feb 08 '21 at 13:39
  • If you want to write an answer I'll validate it. Otherwise I'll post it myself tomorrow for information purposes in case it helps someone. In any case thank you very much. – Chapo Feb 08 '21 at 13:40

0 Answers0