Recursive awaited promise does nothing after resolving

Question

I'm making some requests to the Twitter API, and in order to retrieve tweets I need to perform some recursive method calls as each request will only return a maximum of 100 tweets.

So the process is pretty simple.

Call the function and await it
Perform a http request, await that
If the metadata of the response contains a next_token, cache the results and perform another http request using the next token.
Repeat until the next_token is undefined, at which point resolve the promise with the list of all tweets.

However this isn't working as expected, the recursive http works, however when the else block of the recursive function satisfied and the promise is resolved, nothing happens. Execution doesn't go back to the first function. Everything just seems to spin and do nothing. I've added in breakpoints on every line but nothing causes a breakpoint to trigger either.

Where am I going wrong here?

public async getTweetList(ticker: string): Promise<string[]>{
        let tweets: string[] = [];

        tweets = await this.getAllTweetsRecursively(ticker, null, tweets);
        return tweets;
    }

    public async getAllTweetsRecursively(ticker: string, nextToken: string, tweetList: string[]): Promise<string[]>{
        return new Promise(async (resolve, reject) => {
            let query = `?query=(${ticker})`
            query += this.extraQuery;

            if(nextToken){
                query += this.nextTokenQuery + nextToken
            }

            let res = await axios.default.get(this.url + query, {
                headers: this.headers
            })

            let newNextToken = res.data.meta.next_token;
            if(res.data.data.length > 0 && newNextToken){
                res.data.data.forEach(tweet => {
                    tweetList.push(tweet.text);
                })
                this.getAllTweetsRecursively(ticker, newNextToken, tweetList);
            }
            else {
                resolve(cleanedTweets)
            }
        })
    }

Alternative implementation - same issue

public async getTweetList(ticker: string): Promise<string[]>{
        return new Promise(async (resolve) => {
            let tweets: string[] = [];

            tweets = await this.getAllTweetsRecursively(ticker, null, tweets);
            resolve(tweets);
        })

    }

    public async getAllTweetsRecursively(ticker: string, nextToken: string, tweetList: string[]): Promise<string[]>{
        return new Promise(async (resolve, reject) => {
            let query = `?query=(${ticker})`
            query += this.extraQuery;

            if(nextToken){
                query += this.nextTokenQuery + nextToken
            }

            let res = await axios.default.get(this.url + query, {
                headers: this.headers
            })

            let newNextToken = res.data.meta.next_token;
            if(res.data.data.length > 0 && newNextToken){
                res.data.data.forEach(tweet => {
                    tweetList.push(tweet.text);
                })
                await this.getAllTweetsRecursively(ticker, newNextToken, tweetList);
            }
            else {
                let cleanedTweets: string[] = [];
                tweetList.forEach(tweet => {
                    if(tweet.startsWith("RT")){
                        return;
                    }
                    if(!tweet.toLowerCase().includes("$" + ticker)){
                        return;
                    }
                    cleanedTweets.push(tweet);
                });
                resolve(cleanedTweets)
            }
        })
    }

Casual observation is that only the final promise resolves, the others just end. But I'm a little shaky with manually created promises chained together. — tmdesigned, Mar 04 '21 at 15:30
shouldn't you also `await` the call to `this.getAllTweetsRecursively(ticker, newNextToken, tweetList);` ? — thedude, Mar 04 '21 at 15:34
@thedude I've added a new implementation with both your suggestions. The issue is still there — jm123456, Mar 04 '21 at 15:36
you also don't need the top level promise, just return what you need from your `async` function — thedude, Mar 04 '21 at 15:37
You still haven't resolved tmdesigned's comment, as you need to `resolve` in the case that there is a newNextToken. — samuei, Mar 04 '21 at 15:38
also, also you are not doing anything with whatever is returned from the recursive call — thedude, Mar 04 '21 at 15:38
Either mark a function as `async` and then return a normal value, OR make it a regular function and return a Promise. Don't do both: an `async` function _already wraps the result as a promose_ so your `async function(...) { return new Promise(...) }` is returning a promise that resolves _to a promise_ instead of resolving to actual data. — Mike 'Pomax' Kamermans, Mar 04 '21 at 15:38
Once you are getting results, I suspect you will find that the last page of tweets is missing, since you do nothing with `res.data.data` if `newNextToken` is undefined. — samuei, Mar 04 '21 at 15:40
@samuei Thank you yes, missed that. I've added in an extra block to add the tweets in that condition. — jm123456, Mar 04 '21 at 15:41
@samuei Fixed it, I was missing the resolve in the awaited recursive call as tmdesigned suggested. Works properly now. Thanks all. — jm123456, Mar 04 '21 at 15:42
[Never pass an `async function` as the executor to `new Promise`](https://stackoverflow.com/q/43036229/1048572)! — Bergi, Mar 04 '21 at 22:05

score 1 · Answer 1 · edited Mar 04 '21 at 22:00

The issue is that the inner promise was not resolving in the case where it recursively called itself.

Adding resolve(await this.getAllTweetsRecursively(ticker, newNextToken, tweetList)); fixed the problem.

public async getTweetList(ticker: string): Promise<string[]> {
  let tweets: string[] = [];
  return await this.getAllTweetsRecursively(ticker, null, tweets);
}

public async getAllTweetsRecursively(ticker: string, nextToken: string, tweetList: string[]): Promise<string[]>{
  return new Promise(async (resolve, reject) => {
    let query = `?query=(${ticker})`
    query += this.extraQuery;

    if(nextToken){
      query += this.nextTokenQuery + nextToken
    }

    let res = await axios.default.get(this.url + query, {
      headers: this.headers
    })

    let newNextToken = res.data.meta.next_token;
    if(res.data.data.length > 0 && newNextToken){
      res.data.data.forEach(tweet => {
        tweetList.push(tweet.text);
      })
      resolve(await this.getAllTweetsRecursively(ticker, newNextToken, tweetList));
    }
    else {
      res.data.data.forEach(tweet => {
        tweetList.push(tweet.text);
      })
      let cleanedTweets: string[] = [];
      tweetList.forEach(tweet => {
        if(tweet.startsWith("RT")){
          return;
        }
        if(!tweet.toLowerCase().includes("$" + ticker)){
          return;
        }
        cleanedTweets.push(tweet);
      });
      resolve(cleanedTweets)
    }
  })
}

except you're still mixing `async` and "returning a Promise" so you'll still want to fix that. The `async` keyword is syntactic sugar that makes the function it is applied to wrap its return value in a Promise, so `async function() { return new Promise() }` is the same as `function() { return new Promise(() => new Promise()); }`: don't mix them. Either use `async` and `await`, no promises anywhere, or use promises with `.then()` chaining. — Mike 'Pomax' Kamermans, Mar 04 '21 at 15:50
So I shouldn't use .then() inside of a function which has been awaited? — jm123456, Mar 04 '21 at 21:03
there is no reason to use `.then()` inside an async function, just capture the result using `await`. However, inside a regular function (without the `async` keyword), you can't use `await` (because `await` may only be used inside `async` functions), and so you'll need `.then()` in those cases. — Mike 'Pomax' Kamermans, Mar 04 '21 at 21:58

Mike 'Pomax' Kamermans · Answer 2 · 2021-03-04T22:12:42.447

Remember that async function() {} already defines a functions that return a Promise, because that's what the async keyword does for you. It makes your function body asynchronous by wrapping it in a Promise without you have to write the promise code yourself. And then for convenience, the await keyword is equivalent to Promise.then() without having to write the extraction code yourself.

As such, the following two functions are identical:

function() {
  return new Promise(resolve, reject => {
    resolve(4);
  });
}

async function() {
  return 4;
}

Both return a Promise, both can be awaited inside another async function, and both can be .then() chained, because they are literally the same thing. So in the case of your code, if you want to use async/await you should not also build Promises:

public async getTweetList(ticker: string): Promise<string[]> {
  return await this.getAllTweetsRecursively(ticker, null, []);
}

public async getAllTweetsRecursively(ticker: string, nextToken: string, tweetList: string[]): Promise<string[]>{
    let query = `?query=(${ticker})${this.extraQuery}`;

    if(nextToken){
      query = `${query}${this.nextTokenQuery}${nextToken}`;
    }

    let res = await axios.default.get(this.url + query, {
      headers: this.headers
    })

    let newNextToken = res.data.meta.next_token;
    if(res.data.data.length > 0 && newNextToken){
      res.data.data.forEach(tweet => tweetList.push(tweet.text))
      return await this.getAllTweetsRecursively(ticker, newNextToken, tweetList));
    }

    // no need for "else": the "if" already returns.

    res.data.data.forEach(tweet => tweetList.push(tweet.text))
    let cleanedTweets: string[] = [];
    tweetList.forEach(tweet => {
      if(tweet.startsWith("RT")) return;
      if(!tweet.toLowerCase().includes("$" + ticker)) return;
      cleanedTweets.push(tweet);
    });

    return cleanedTweets
  })
}

Your code contains async getAllTweetsRecursively(...) { return new Promise(async (resolve, reject) => { ... })) which will now return THREE nested promises, because those async keywords make this:

getAllTweetsRecursively(...) {
  return new Promise(resolve, reject => {
    return new Promise(async (resolve, reject) => {
      ...
    }))
  });
}

which because of that async further unrolls to:

getAllTweetsRecursively(...) {
  return new Promise(resolve, reject => {
    return new Promise((resolve, reject) => {
      return new Promise(resolve, reject => {
        resolve(...)
      })
    }))
  });
}

this way lies madness =)

Recursive awaited promise does nothing after resolving

2 Answers2