Coffescript Nested Do Loop / Async Nested Loop

Question

I want to save into two collections in my mongoDB. This operations are async so I use for and do in coffee.

for machine in machines
  do(machine) ->
    //if machine does not exist
      for part in machine.parts
        do(part) ->
          //if not part not exists --> save
          //push part to machine parts list
      //save machine

The machine parts are empty later in the db. How can I make the first do loop wait for the second do loop to finish?

EDIT Real Code Example:

  recipeModel = require('../models/recipe.model')
  ingredientModel = require('../models/ingredient.model')

  #Save Recipe into the database
  async.map recipes, (recipe, next) ->
    recipeDBObject = new recipeModel()
    recipeDBObject.href = recipe.href
    recipeDBObject.ingredients = []
    recipeModel.find({ href: recipe.href }, (err, recipeFound) ->
      return next err if err
      return next null, recipeFound if recipeFound.length > 0

      recipeDBObject.title = recipe.title
      ingredientsPushArray = []
      console.log recipe.href

      async.map recipe.zutaten, (ingredient, cb) ->
        #Save all ingredients
        ingredient.idName = ingredient.name.replace(/[^a-zA-Z0-9]+/gi, "").toLowerCase()
        ingredientModel.find({ idName: ingredient.idName }, (err, ingredientFound) ->
          return next err if err
          if ingredientFound.length >0
            ingredientDBObject = ingredientFound[0]
          else
            ingredientDBObject = new ingredientModel()

          ingredientDBObject.name = ingredient.name
          ingredientDBObject.save()

          recipeDBObject.ingredients.push({"idName":ingredient.idName, "name":ingredient.name, "amount":ingredient.amount})
          return cb(null, true)
        )
      recipeDBObject.ingredients = ingredientsPushArray
      recipeDBObject.save()
      return next(null, true)
    )

I still don't get it working. Recipes are saved, node builds the ingredients array but it neither saves the ingredients nor does it save the array into the recipes.

EDIT 2:

  async.map recipes,
    (recipe, next) ->
      recipeDBObject = new recipeModel()
      recipeDBObject.href = recipe.href
      recipeDBObject.ingredients = []
      recipeModel.find({ href: recipe.href }, (err, recipeFound) ->
        return next err if err
        return next null, recipeFound if recipeFound.length > 0

        recipeDBObject.title = recipe.title
        ingredientsPushArray = []

        ingredientsArray = []
        async.map recipe.zutaten,
          (ingredient, cb) ->
            #Save all ingredients
            ingredient.idName = ingredient.name.replace(/[^a-zA-Z0-9]+/gi, "").toLowerCase()
            ingredientModel.find({ idName: ingredient.idName }, (err, ingredientFound) ->
              return next err if err
              ingredientsArray.push({"idName":ingredient.idName, "name":ingredient.name, "amount":ingredient.amount})
              if ingredientFound.length >0
                return cb(null, true)
              else
                ingredientDBObject = new ingredientModel()
                ingredientDBObject.name = ingredient.name
                ingredientDBObject.idName = ingredient.idName
                ingredientDBObject.save((err) ->
                  #console.log "some erros because required is empty" if err
                  return cb err if err
                  #console.log "ingredient saved"
                  return cb(null, true)
                )
          (err, ingredientsArray) ->
            console.log "This is never logged"
            return err if err
            recipeDBObject.ingredients = ingredientsArray
            recipeDBObject.save((err)->
              return err if err
              return next(null, true)
          )
        )
      )
    (err) ->
      console.log "show me the errors: ", err if err

Now the ingredients are saved but the recipes aren't.

Interesting ressources: http://www.hacksparrow.com/managing-nested-asynchronous-callbacks-in-node-js.html

You're not waiting for `.save()` methods to complete and not handling callbacks from `async.map`. Since you're not actually handling your async stuff, your example works pretty much like synchronous for loop did. — Leonid Beschastny, Apr 01 '15 at 10:31
I mean, you just calling `obj.save()` and then synchronously invoking `asnc` callback, instead of passing it to mongoose like `obj.save next`. — Leonid Beschastny, Apr 01 '15 at 10:36
And since you're not handling `async.map` callback, you're trying to save `recipeDBObject` before any of `recipe.zutaten` objects have been processed. You should move this part of your code to the `async.map` callback. — Leonid Beschastny, Apr 01 '15 at 10:38
You're not handling `async` errors correctly, e.g. you're calling `return next err if err` inside of `(ingredient, cb) ->` iterator function instead of `return cb err if err`, and returning an error `return err if err` instead of passing it to the next handler `return next err if err`. Though, I can't see why `console.log` is newer called in your example (unless an actual error occurred in mongoose call). — Leonid Beschastny, Apr 01 '15 at 12:10
Pushing data to a shared array is not a very good way to pass it to the final callback. Since you're using `async.map` and not `async.forEach`, you could simply pass ingredients to the `cb` callback (as its second argument). Otherwise, the order of elements in `ingredientsArray` array will not be the same as it was in original array. Well, and you're shadowing shared `ingredientsArray` variable with a local one, anyway. — Leonid Beschastny, Apr 01 '15 at 12:14

Leonid Beschastny · Accepted Answer · 2015-04-01T10:49:28.980

2

The easiest way is to use some module for for managing asynchronous control flow, for example

async
promise-based solutions (e.g. when, bluebird, Q)
co for ES6 generator-based control flow

Here are some simple examples.

Using `async.map`

async = require 'async'

async.map machines,
  (machine, next) ->
    # Process single machine object
    Machine.findById machine._id, (err, found) ->
      return next err if err # return error immediately
      return next null, found if found # return the object we found
      async.map machine.parts,
        (part, cb) ->
          # Save part to DB and call cb callback afterward
          Part.create part, cb
        (err, parts) ->
          return next err if err # propagate error to the next handler
          # All parts have been saved successfully
          machine.parts = parts
          # Save machine to DB and call next callback afterward
          Machine.create machine, next
  (err, machines) ->
    if err
      # Something went wrong
    else
      # All machine objects have been processed successfully

Using promises and `when` module

When = require 'when'

machines_to_save = When.filter machines, ({_id}) ->
  Machine.findById(_id).then (found) -> not found
When.map machines_to_save, (machine) ->
  When.map machine.parts, (part) ->
    Part.create part
  .then (parts) ->
    machine.parts = parts
    Machine.create machine
.then (saved_machines) ->
  # All machines are saved
.otherwice (err) ->
  # Something went wrong

edited Apr 01 '15 at 10:49

answered Apr 01 '15 at 09:04

Leonid Beschastny

50,364
10
118
122

Thanks for your answer. I guess this will be the solution but I still don't get it running, could you take a brief look at the code above. – Andi Giga Apr 01 '15 at 10:22
@AndiGiga I changed code formatting in my `async` example to make it easier for for you to understand what's going on there. – Leonid Beschastny Apr 01 '15 at 10:44
@AndiGiga also added a lot of comments to explain each step for you. – Leonid Beschastny Apr 01 '15 at 10:50
That is what I think I understood. #1) If I call a second function after a first function, like you did with `(part, cb) ->` & `(err, parts) ->`), then the first function is executed on every item of the Array, then the second function gets executed? #2) If there is a return in the first function, the second function still gets executed? #3) You can pass params between the first and the second function by just declaring them in the second functions arguments? #4) The async.map only continues with the next element when the callback is resolved. E.g. what you do with `return next err if err`? – Andi Giga Apr 01 '15 at 11:36
I also edited my code. Sorry for not understanding this, but I guess at some point it will click ;-) – Andi Giga Apr 01 '15 at 11:39
`If I call a second function after a first function` you're not calling them, you're passing them to `async.map` as arguments. – Leonid Beschastny Apr 01 '15 at 11:47
`If there is a return in the first function, the second function still gets executed?` `asyc` works with callbacks, so all return values are ignored. It's just a way to break function execution somewhere in the middle. The final callback function will be executed after you'll call all `next` callbacks. – Leonid Beschastny Apr 01 '15 at 11:49
`You can pass params between the first and the second function by just declaring them in the second functions arguments?` nope, `async.map` calls its final callback with two arguments. First argument is an error (if one occured), and the second one is the new mapped array. `async.map` expects that you provide an error and new mapped element on each iteration through the `next` callback. See [`async.map` docs](https://github.com/caolan/async#map). for more info – Leonid Beschastny Apr 01 '15 at 11:53
`The async.map only continues with the next element when the callback is resolved.` actually, it executes all tasks in parallel (well, [not actually in parallel, but sort of](http://stackoverflow.com/questions/19023977/async-js-is-parallel-really-parallel)). But when an error occurs, `async` immediately calls its final callback ignoring all unfinished tasks. – Leonid Beschastny Apr 01 '15 at 11:56
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/74259/discussion-between-leonid-beschastny-and-andi-giga). – Leonid Beschastny Apr 01 '15 at 11:57
Ok thx I guess I have it now. The doc was not very useful for me actually I took a look at it before, but it has basically no examples. You clarified it a lot thx. – Andi Giga Apr 01 '15 at 12:42
1

Actually I noticed I have to use async.mapSeries otherwise he does not wait for the callback which causes duplicate entry errors. – Andi Giga Apr 02 '15 at 07:10
@AndiGiga Yes, `async.map` runs all its task all at once. I mentioned it in one of my comments, but I haven't thought that it'll be a problem in your case. – Leonid Beschastny Apr 02 '15 at 07:17

Coffescript Nested Do Loop / Async Nested Loop

1 Answers1

Using async.map

Using promises and when module

Using `async.map`

Using promises and `when` module