Best way to call an asynchronous function within map?

Question

I'm mapping over an array and for one of the return values of the new object, I need to make an asynchronous call.

var firebaseData = teachers.map(function(teacher) {
  return {
    name: teacher.title,
    description: teacher.body_html,
    image: urlToBase64(teacher.summary_html.match(/src="(.*?)"/)[1]),
    city: metafieldTeacherData[teacher.id].city,
    country: metafieldTeacherData[teacher.id].country,
    state: metafieldTeacherData[teacher.id].state,
    studioName: metafieldTeacherData[teacher.id].studioName,
    studioURL: metafieldTeacherData[teacher.id].studioURL
  }
});

The implementation of that function will look something like

function urlToBase64(url) {
  request.get(url, function (error, response, body) {
    if (!error && response.statusCode == 200) {
      return "data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64');
    }
  });
}

I'm not clear what's the best approach to do this... promises? Nested callbacks? Use something in ES6 or ES7 and then transpile with Babel?

What's the current best way to implement this?

Answer 1

update in 2018: Promise.all async function within map callback is easier to implement:

    let firebaseData = await Promise.all(teachers.map(async teacher => {
        return {
            name: teacher.title,
            description: teacher.body_html,
            image: await urlToBase64(teacher.summary_html.match(/src="(.*?)"/)[1]),
            city: metafieldTeacherData[teacher.id].city,
            country: metafieldTeacherData[teacher.id].country,
            state: metafieldTeacherData[teacher.id].state,
            studioName: metafieldTeacherData[teacher.id].studioName,
            studioURL: metafieldTeacherData[teacher.id].studioURL
        }
    }));


async function urlToBase64(url) {
  return request.get(url, function (error, response, body) {
    if (!error && response.statusCode == 200) {
      return "data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64');
    }
  });
}

Edit@2018/04/29: I put the general example for everyone:

Edit@2019/06/19 : async/await should have try/catch to handle errors to make your process continue working in case of some of requests failed.

let data = await Promise.all(data.map(async (item) => {
      try {
      item.fetchItem = await fetchFunc(item.fetchParams);

      return item; 
      } catch(error) {
         return {...item, error } ;
      }
  }));
  /* we can filter errors in data and retry later 
   * eg:
   * const errorItems = data.filter(item => !!item.error)
   */
 

Answer 2

One approach is Promise.all (ES6).

This answer will work in Node 4.0+. Older versions will need a Promise polyfill or library. I have also used ES6 arrow functions, which you could replace with regular functions for Node < 4.

This technique manually wraps request.get with a Promise. You could also use a library like request-promise.

function urlToBase64(url) {
  return new Promise((resolve, reject) => {
    request.get(url, function (error, response, body) {
      if (!error && response.statusCode == 200) {
        resolve("data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64'));
      } else {
        reject(response);
      }
    });
  })
} 

// Map input data to an Array of Promises
let promises = input.map(element => {
  return urlToBase64(element.image)
    .then(base64 => {
      element.base64Data = base64;
      return element;
    })
});

// Wait for all Promises to complete
Promise.all(promises)
  .then(results => {
    // Handle results
  })
  .catch(e => {
    console.error(e);
  })

Answer 3

In 2020 we now have the for await...of syntax of ECMAScript2021 that significantly simplifies things:

So you can now simply do this:

//return an array of promises from our iteration:
let promises = teachers.map(async m => {
   return await request.get(....);
});

//simply iterate those
//val will be the result of the promise not the promise itself
for await (let val of promises){
   ....
}

Answer 4

You can use async.map.

var async = require('async');

async.map(teachers, mapTeacher, function(err, results){
  // results is now an array of stats for each file
});

function mapTeacher(teacher, done) {
  // computing stuff here...
  done(null, teacher);
}

note that all teachers will be processed in parallel - you can use also this functions:

mapSeries(arr, iterator, [callback]) maps one by one

mapLimit(arr, limit, iterator, [callback]) maps limit at same time

Answer 5

I had a similar problem and found this to be easier (I'm using Kai's generic template). Below, you only need to use one await. I was also using an ajax function as my async function:

function asyncFunction(item) {
    return $.ajax({
        type: "GET",
        url: url,
        success: response => {
            console.log("response received:", response);
            return response;
        }, error: err => {
            console.log("error in ajax", err);
        }
    });
}

let data = await Promise.all(data.map(item => asyncFunction(item)));

Answer 6

Try amap(), the asynchronous map function below:

async function amap(arr,fun) {
    return await Promise.all(arr.map(async v => await fun(v)))
}

Or, written in a more concise way:

let amap = async (arr,fun) => await Promise.all(arr.map(async v => await fun(v)))

Usage:

let arr = await amap([1,2,3], async x => x*2)
console.log(arr)   // [2, 4, 6]

Answer 7

I am using an async function over the array. And not using array.map, but a for function. It's something like this:

const resultingProcessedArray = async function getSomeArray() {
    try {
      let { data } = await axios({url: '/myUrl', method:'GET'}); //initial array
      let resultingProcessedArray = [];
      for (let i = 0, len = data.items.length; i < len; i++) {
        let results = await axios({url: `/users?filter=id eq ${data.items[i].someId}`, method:'GET'});
        let domainName = results.data.items[0].domainName;
        resultingProcessedArray.push(Object.assign(data.items[i], {domainName}));
      }
      return resultingProcessedArray;
    } catch (err) {
      console.error("Unable to fetch the data", err);
      return [];
    }
};

Answer 8

I had to write this, for the sake of convenience. Otherwise, I might need https://github.com/mcollina/make-promises-safe

export async function mapAsync<T, U>(
  arr: T[], 
  callbackfn: (value: T, index: number, array: T[]) => Promise<U>, 
  thisArg?: any
) {
  return await Promise.all(arr.map(async (value, index, array) => {
    try {
      return await callbackfn(value, index, array);
    } catch(e) {
      throw e;
    }
  }, thisArg));
}

Answer 9

For production purposes you probably want to use a lib like lodasync, you should not reinvent the wheel:

import { mapAsync } from 'lodasync'

const result = await mapAsync(async(element) => {
  return 3 + await doSomething(element)
}, array)

It uses promises, has no dependencies, and is as fast as it gets.

Answer 10

By using Promise.all you can make map and forEach work with async functions (i.e. Promises).

To make filter, some and every work you can first use an async map (that in turn uses Promise.all) and then go through the true/false values and synchronously do the filtering/evaluation.

To make reduce and reduceRight work with async functions you can wrap the original function in a new one that waits for the accumulator to resolve.

Using this knowledge it is possible to modify the original array methods in a way so that they continue to work "as usual" with normal/synchronous functions but will also work with async functions.

// a 'mini library' (save it somewhere and import it once/project)
(() => {
  let AsyncFunction = Object.getPrototypeOf(async e => e).constructor;
  ['map', 'forEach'].forEach(method => {
    let orgMethod = Array.prototype[method];
    Array.prototype[method] = function (func) {
      let a = orgMethod.call(this, func);
      return func instanceof AsyncFunction ? Promise.all(a) : a;
    };
  });
  ['filter', 'some', 'every'].forEach(method => {
    let orgMethod = Array.prototype[method];
    Array.prototype[method] = function (func) {
      if (func instanceof AsyncFunction) {
        return (async () => {
          let trueOrFalse = await this.map(func);
          return orgMethod.call(this, (_x, i) => trueOrFalse[i]);
        })();
      }
      else {
        return orgMethod.call(this, func);
      }
    };
  });
  ['reduce', 'reduceRight'].forEach(method => {
    let orgMethod = Array.prototype[method];
    Array.prototype[method] = function (...args) {
      if (args[0] instanceof AsyncFunction) {
        let orgFunc = args[0];
        args[0] = async (...args) => {
          args[0] = await args[0];
          return orgFunc.apply(this, args);
        };
      }
      return orgMethod.apply(this, args);
    };
  });
})();

// AND NOW:

// this will work
let a = [1, 2, 3].map(x => x * 3); // => [3, 6, 9]
let b = [1, 2, 3, 4, 5, 6, 7].filter(x => x > 3); // [4, 5, 6, 7]
let c = [1, 2, 3, 4, 5].reduce((acc, val) => acc + val); // => 15

// this will also work
let x = await [1, 2, 3].map(async x => x * 3);
let y = await [1, 2, 3, 4, 5, 6, 7].filter(async x => x > 3);
let z = await [1, 2, 3, 4, 5].reduce(async (acc, val) => acc + val);

Answer 11

If you'd like to map over all elements concurrently:

function asyncMap(arr, fn) {
  return Promise.all(arr.map(fn));
}

If you'd like to map over all elements non-concurrently (e.g. when your mapping function has side effects or running mapper over all array elements at once would be too resource costly):

Option A: Promises

function asyncMapStrict(arr, fn) {
  return new Promise((resolve) => {
    const result = [];
    arr.reduce(
      (promise, cur, idx) => promise
        .then(() => fn(cur, idx, arr)
          .then((res) => {
            result.push(res);
          })),
      Promise.resolve(),
    ).then(() => resolve(result));
  });
}

Option B: async/await

async function asyncMapStrict(arr, fn) {
  const result = [];

  for (let idx = 0; idx < arr.length; idx += 1) {
    const cur = arr[idx];

    result.push(await fn(cur, idx, arr));
  }

  return result;
}

Answer 12

The best way to call an async function within map is to use a map created expressly for async functions.

For a function to be async, it must return a Promise.

function urlToBase64(url) {
  return new Promise((resolve, reject) => {
    request.get(url, function (error, response, body) {
      if (error) {
        reject(error)
      } else if (response && response.statusCode == 200) {
        resolve(
          "data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64');
        )
      } else {
        reject(new Error('invalid response'))
      }
    });
  })
}

Now, we can map:

const { pipe, map, get } = require('rubico')

const metafieldTeacherData = {} // { [teacher_id]: {...}, ... }

const parseTeacher = teacher => ({
  name: teacher.title,
  description: teacher.body_html,
  image: urlToBase64(teacher.summary_html.match(/src="(.*?)"/)[1]),
  city: metafieldTeacherData[teacher.id].city,
  country: metafieldTeacherData[teacher.id].country,
  state: metafieldTeacherData[teacher.id].state,
  studioName: metafieldTeacherData[teacher.id].studioName,
  studioURL: metafieldTeacherData[teacher.id].studioURL
})

const main = async () => {
  const teachers = [] // array full of teachers
  const firebaseData = await map(pipe([
    parseTeacher,
    get('studioURL'),
    urlToBase64,
  ]))(teachers)
  console.log(firebaseData) // > ['data:application/json;base64,...', ...]
}

main()

rubico's map worries about Promise.all so you don't have to.

Answer 13

Using IIFE and Promise.all, can make a simple use cases.

await Promise.all(arr.map(el=>(async _=>{
    // some async code       
})()))

This IIFE can return a promise which is used as the return value for map function.

(async _=>{
    // some async code       
})()

So arr.map will return a list of promise to Promise.all to handle.

Example

const sleep = (ms) => {
  return new Promise((resolve, reject) => {
    setTimeout(_ => {
      resolve()
    }, ms)
  });
}

await Promise.all([1000,2000,3000].map(el=>(async _=>{
    await sleep(el)
    console.log(el) 
    return el  
})())) 

Answer 14

Here is a simple function that will let you choose to await each mapping operation (serial) or run all mappings in parallel.

The mapper function need not return a promise either.

async function asyncMap(items, mapper, options = {
  parallel: true
}) {
  const promises = items.map(async item => options.parallel ? mapper(item) : await mapper(item))
  return Promise.all(promises)
}

Typescript Version

async function asyncMap<I, O>(items: I[], mapper: (item: I) => O, options = {
  parallel: true
}): Promise<O[]> {
  const promises = items.map(async item => options.parallel ? mapper(item) : await mapper(item))
  return Promise.all(promises)
}

A working snippet for science

async function asyncMap(items, mapper, options = {
  parallel: true
}) {
  const promises = items.map(async item => options.parallel ? mapper(item) : await mapper(item))
  return Promise.all(promises)
}

// A test to multiply number by 2 after 50 milliseconds
function delay(num) {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(num * 2)
    }, 50)
  })
}

;
(async() => {
  const data = [1, 2, 3, 4]
  const resParallel = await asyncMap(data, it => delay(it))
  const resSerial = await asyncMap(data, it => delay(it), {
    parallel: false
  })
  console.log(data)
  console.log(resParallel)
  console.log(resSerial)

})();