How to execute code while waiting for a promise to finish

Question

I have an express application where I have a generator function that needs aprox. 5 minutes for processing a lot of data. Unfortunately I can not optimize that function.
Express automatically times out after 2 minutes and I do not want to alter that only for this specific function. I figured that maybe if I made periodically a res.write() call, the 2 minute rule does not apply.

My question:
How can I execute a res.write('Something') every X seconds while waiting for the other function to terminate ?

I want it to do something like the following, I hope you get the idea.

    function main() {
      co(function* () {
        const something = yield promise(); // function that needs a lot of time
        const doWhileWaiting = setTimeout(() => {
          if (!something) {
            // Print this while waiting for the value of something
            console.log('Waiting for something ... ');
          } else {
            console.log(something);
            clearInterval(doWhileWaiting);
          }
        }, 5000);
      });
    }

something like this? http://stackoverflow.com/questions/9751711/streaming-http-responses-with-nodejs — Kevin B, Mar 20 '17 at 20:51
Do you want to make the user wait for the full five minutes? You can send a response like "Thanks, your data will be ready in five minutes." and provide a notification via webhook, web socket, push notification, etc. or just give them a link to where they can check when it's ready. — Explosion Pills, Mar 20 '17 at 20:53
This is for an application, eventually the server will execute a script once a day that calls this function. The problem is that because of the yield, everything after ´´´something´´´ won't be executed until it finishes. Is there a way to execute it in background an get a notification when it's finished ? — Felix, Mar 20 '17 at 21:03
What exactly do you mean by "*the server will execute a script once a day*"? Do you mean another server makes a HTTP request to your endpoint? Otherwise I don't see what express has to do with this. — Bergi, Mar 20 '17 at 21:28

score 0 · Accepted Answer · answered Mar 20 '17 at 22:04

You should not have such a heavy, blocking function in your api itself. Much better would be to have another process handle this sort of computing for you as node is single threaded and this will block all other computation. There sure are ways to do this in one thread/process by adding timeout in your computing functions, but that's not very nice style. As i don't know what your function is doing, it's hard to give a solid answer what's best in your case.

As you say its executed once a day, you could do a cronjob trigger another process. If you need to communicate to your api process, you might want to look at child processes. https://nodejs.org/api/child_process.html

score 0 · Answer 2 · answered Mar 20 '17 at 22:59

This is not very hard to implement:

const secret = Symbol();
const delay = ms => new Promise(r => setTimeout(() => r(secret), ms));

async waitFor(promise) {
    while(true) {
        const racer = await Promise.race(promise, delay(1));
        if(racer !== secret) return; // our promise won!
        await delay(100);
   }
}

At this point, you'd want to add an action to run whenever it's not done yet.

async waitFor(promise, action) {
    while(true) {
        const racer = await Promise.race(promise, delay(1));
        if(racer !== secret) return; // our promise won!
        await action(); // support passing a promise action.
        await delay(100);
   }
}

// will write to `res` every 100 ms while `yourPromise is not resolved.
waitFor(yourPromise, () => res.write("..."));

Note that fundamentally you should place the extra work in a work queue and use a separate process to handle it. Node really isn't cut for this kind of workload. I recommend checking out the other answer for alternatives.

How to execute code while waiting for a promise to finish

2 Answers2