fetch items from database in steps and perform operations sequentially

Question

I have a database with a lot of items and I need to perform an action on each of these items.Unfortunatly, I have to run these actions sequentially and also delay every action to avoid rate limit.

My approach does not wait for the previous actions to be finished and I end up getting rate limited. What do I have to change in order for it to run sequentially?

  setInterval(async () => {
      await this.processQueue();
    }, 1500)

 private async processQueue() {
    try {
      //Only 3 requests per second allowed by the API so I only take 3 items from the database on every call
      const bids = await getRepository(Bid).find({ order: { created_at: "ASC" }, take: 3, skip: 0 })

      if (bids) {
        for (const bid of bids) {
          //check if active order exists
          const activeOrder = await this.accountService.hasActiveOrder(bid.tokenId, bid.tokenAddress, 0);

          if (!activeOrder) {
            //perform async functions with that db item
            const startTime = Date.now();
            await this.placeBid(bid);
            //delete from database so the next call to processQueue does not return the same itemsagain
            await getRepository(Bid).delete({ id: bid.id })
            const endTime = Date.now() - startTime;
          }
        }
      }

    } catch (error) {
      console.error("TradingService processQeueu", error.message);
    }
  }

T.J. Crowder · Accepted Answer · 2022-01-19T09:38:23.137

1

There's no coordination between your interval timer and the work being done by the processQueue function. Passing an async function to setInterval as its callback is misleading and not useful; setInterval doesn't use the callback's return value, so the promise it returns isn't used for anything.

Instead, the minimal change would be to use something that does wait for processQueue to finish, perhaps a chained series of setTimeout callbacks:

function processAndWait() {
    processQueue().then(() => setTimeout(processAndWait, 1500));
    // (Normally I'd have a `catch` call as well, but `processQueue`
    // ensures it never rejects its promise)
}

Note that that waits 1500ms after the queue has been processed. That's probably overkill, if the API allows up to three calls per second. You could probably trim it to 1000ms.

Or if this is a method (public or private) in a class:

processAndWait() {
    processQueue().then(() => {
        this.queueTimer = setTimeout(() => this.processAndWait(), 1500);
    });
    // (Normally I'd have a `catch` call as well, but `processQueue`
    // ensures it never rejects its promise)
}

Note that I added something saving the timer handle to a property, so you can use clearTimeout(this.queueTimer) to stop the process.

edited Jan 19 '22 at 09:38

answered Jan 19 '22 at 09:08

T.J. Crowder

1,031,962
187
1,923
1,875

and what would I need to do in order to processAndWait multiple times then? As far as I understand, your code only performs processQueue once. – BitQueen Jan 19 '22 at 09:15
@BitQueen - What makes you think that? Look at the `then` handler's code. It sets up a timed callback to `processAndWait`, which will process the queue, wait for that to finish, and the set up a timed callback to `processAndWait`, which will... That's why I called it *"a chained series of `setTimeout` callbacks"*. – T.J. Crowder Jan 19 '22 at 09:16
1

oh my bad I am sorry. Thank you for your recommendation I will try that! – BitQueen Jan 19 '22 at 09:18
I have changed it to this private processAndWait() { this.processQueue().then(() => setTimeout(this.processAndWait, 1200)); } and then call within update() this.processAndWait() which returns an error "processQueue is not a function" – BitQueen Jan 19 '22 at 09:27
@BitQueen - See [this question's answers](http://stackoverflow.com/questions/20279484/how-to-access-the-correct-this-context-inside-a-callback) (and perhaps [this one](https://stackoverflow.com/questions/3127429/how-does-the-this-keyword-work)), you're not preserving `this` when `setTimeout` calls the method. – T.J. Crowder Jan 19 '22 at 09:32
@BitQueen - I've updated the answer a bit. – T.J. Crowder Jan 19 '22 at 09:38
thank u now its working. I tried using let self = this; and using that for calling which also didnt do the trick. Ur last edit worked for me. Thank u – BitQueen Jan 19 '22 at 09:46
@BitQueen - Sounds like you were doing the `self` trick in the wrong place. But there's never any need for the `self` trick anymore, we have arrow functions now (or `.bind`). See the answers there that use those (since this is likely to come up for you again). Happy coding! – T.J. Crowder Jan 19 '22 at 09:48

fetch items from database in steps and perform operations sequentially

1 Answers1