nodejs download image from url synchronously

Question

I am trying to build a small node app it calls an api which returns an array of urls which point to image blob png files.

I am then trying to loop over the array and download the files using a utility function. I need this to work synchronously. Once the downloads are complete I then want to fire an additional function.

I started off using some asynchronous code which I took from here: https://sabe.io/blog/node-download-image

The async code in my utils file looked like this:

import { promises as fs } from "fs";
import fetch from "node-fetch";

const downloadFile = async (url, path, cb) => {
  const response = await fetch(url);
  const blob = await response.blob();
  const arrayBuffer = await blob.arrayBuffer();
  const buffer = Buffer.from(arrayBuffer);
  fs.writeFile(path, buffer);
  cb();
}

export { downloadFile };

I have tried to convert it to be purely synchronous using this code:

import fs from "fs";
import fetch from "node-fetch";

const downloadFile = (url, path, cb) => {
    const response = fetch(url);
    const blob = response.blob();
    const arrayBuffer = await blob.arrayBuffer();
    const buffer = Buffer.from(arrayBuffer);
    fs.writeFileSync(path, buffer);
    cb();
  }

export { downloadFile };

Then in my index.js file I am using it like so:

import { downloadFile } from './utils/downloadFiles.js';

let imagesArray = [];
let newImageNames = [];

imagesArray.forEach((item, index) => {
   const fileName = `${prompt}__${index}_${uuid.v4()}.png`;
   const filePath = path.join('src', 'images'); 
   newImageNames.push(fileName);
   downloadFile(item, filePath, fileDownloadCallback);
});

processDataCallback(); // This is the function which is being fired before the previous downloadFile functions have finished processing.

const fileDownloadCallback = () => {
   console.log(`File download callback`);
}

My images array is being populated and looks like this as an example:

data: [
  {
    url: 'https://someurl.com/HrwNAzC8YW/A%3D'
  },
  {
    url: 'https://someurl.com/rGL7UeTeWTfhAuLWPg%3D'
  },
  {
    url: 'https://someurl.com/xSKR36gCdOI3/tofbQrR8YTlN6W89DI%3D'
  },
  {
    url: 'https://someurl.com/2y9cgRWkH9Ff0%3D'
  }
]

When I try and use the synchronous method I get this error TypeError: response.blob is not a function. This function does work when using it asynchronously, but then it is firing my next function before the image downloads have finished.

I have tried several iterations, first off using createWriteStream and createWriteStreamSync (which I believe are deprecated). So switched to fileWrite. I also tried using a synchronous fileWriteSync inside the async function, but still no dice. The other issue is that fetch works asynchronously, so I still don't know how to wire this up to only work synchronously. I was also wondering If I could chain a then onto the end of my fileDownload util function.

All of my code is in github, so I can share a url if required. Or please ask for more explanation if needed.

Is there something equivalent to jsfiddle for Node? If so I am more than happy to try and make a demo.

Any help greatly appreciated.

Answer 1

We can leave the original async downloadFile util alone (though there's a little room for improvement there).

In the index file...

import { downloadFile } from './utils/downloadFiles.js';

let imagesArray = [];
let newImageNames = [];

// I'm a little confused about how we get anything out of iterating an empty array
// but presuming it get's filled with URLs somehow...
const promises = imagesArray.map((item, index) => {
   const fileName = `${prompt}__${index}_${uuid.v4()}.png`;
   const filePath = path.join('src', 'images'); 
   newImageNames.push(fileName);
   // we can use the callback for progress, but we care about finishing all
   return downloadFile(item, filePath, () => {
     console.log('just wrote', filePath);
   });
});

Promise.all(promises).then(() => {
  console.log('all done')
})

Answer 2

Array.forEach doesnt work with async code.

Convert your code into a for...of or for code and it will works.

Also. You don't use callbacks with async/await code.

Your code will look like this:

let index = 0;
for(const item of imageArray) => {
   const fileName = `${prompt}__${index}_${uuid.v4()}.png`;
   const filePath = path.join('src', 'images'); 
   newImageNames.push(fileName);
   downloadFile(item, filePath);
   fileDownloadCallback();
   index++;
});