Exchange sync for loop to async approach?

Question

I have this code where I try to cache neccesary memories before using them to do HTTP requests with a library. I cache them before to make the code faster when it actually do the requests which is critical to go as fast as possible.

The code is working but now it does the requests in a "synchronous" manner.

I beleive the problem code is the below line:

for (i = 0; i < exchanges.length; i++)

I am not sure what is the best/fastest approach to run the above for loop async?

'use strict';
const ccxt = require('ccxt');
const fs = require('fs');
const path = require('path');

//Cache some memories first
var exchangesArray = [];
var exchangesArray2 = [];
(async () => {
  const allexchanges = ccxt.exchanges.filter((id) => !['coinmarketcap', 'theocean'].includes(id))
        .map(async (id) => {
            const Exchange = ccxt[id];
            const exchange = new Exchange({ enableRateLimit: true });
            if (exchange.has['fetchTickers']) {

                exchangesArray.push(exchange);
                exchangesArray2.push(id);
            }
        });

    await Promise.all(allexchanges);
})();



//The cached memories
const exchanges = exchangesArray; //Holds exchanges object
const exchangesnames = exchangesArray2; //Holds exchanges name
var i;

//Use cached memories to do the "fetchTickers()" as fast as possible now
(async () => {
    console.log(`start`);
    const start = Date.now();


    //What is the fastest way to make this for loop async/await to run in parallel?
    for (i = 0; i < exchanges.length; i++) {

        // pop one exchange from the array
        const exchange = exchanges[i]
        const exchangename = exchangesnames[i]

        try {
            const tickers = await exchange.fetchTickers();
            const dumpFile = path.join(__dirname, 'tickers', `${exchangename}Tickers.json`);
            await fs.promises.writeFile(dumpFile, JSON.stringify(tickers));
        } catch (e) {
            console.error(e);
        }
    }


    // wait for all of them to execute or fail
    await Promise.all(exchanges)


    const end = Date.now();
    console.log(`Done in ${(end - start) / 1000} seconds`);
})();

Possible duplicate of [Using async/await with a forEach loop](https://stackoverflow.com/questions/37576685/using-async-await-with-a-foreach-loop) — Bonjov, Feb 22 '19 at 21:30
I was thinking the same but are not sure if it is the same problem. I think this is specifically a different problem. — Andreas, Feb 22 '19 at 21:39

score 0 · Answer 1 · answered Feb 22 '19 at 21:56

0

Take a look at the for await...of syntax.

answered Feb 22 '19 at 21:56

john-goldsmith

2,969
1
19
18

Yes, I only wonder if I use: for await (let exchange of exchanges) { } I think I have another problem. I do get the exchange there but I have another array that holds the exchangesnames. How will I get both in that loop? – Andreas Feb 22 '19 at 22:00
Did you try nested `for...of` or `for await...of` loops? – john-goldsmith Feb 22 '19 at 22:11
I am not sure how to do nested loops in this case as it is 2 different arrays. I can't do it like this. I do not get the Same index from each loop in one iteration? Can nested loops really work here? for await (let exchange of exchanges) { for await (let exchangename of exchangesnames) { } } – Andreas Feb 22 '19 at 22:22

score 0 · Accepted Answer · answered Feb 22 '19 at 22:19

Try this. Basically you create an array of promises for each operation to save your tickers. Then use Promise.all to wait for all of the ticker processes to resolve.

//What is the fastest way to make this for loop async/await to run in parallel?
    var tickersPromises = []
    for (i = 0; i < exchanges.length; i++) {

        // pop one exchange from the array
        const exchange = exchanges[i]
        const exchangename = exchangesnames[i]

        try {
            let tickerProcessing = new Promise(async (resolve) => {
                // probably do a try catch in here
                const tickers = await exchange.fetchTickers();
                const dumpFile = path.join(__dirname, 'tickers', `${exchangename}Tickers.json`);
                await fs.promises.writeFile(dumpFile, JSON.stringify(tickers));
                resolve()
            })
            tickersPromises.push(tickerProcessing)

        } catch (e) {
            console.error(e);
        }
    }


    // wait for all of them to execute or fail
    await Promise.all(tickersPromises)

From a high level, if you really want to make this code faster stop writing the response to a file and store them in memory using an object or a dictionary library. That said the web latency is going to your slowest part.

I'm guessing these are crypto exchanges. Keep in mind the tickers are often delayed from the crypto exchanges when using the REST api. You are better off using the websocket interface many of the exchanges offer to push you the latest ticker information as soon as it is available to minimize latency.

That actually worked. I will remember how to do new Promise(async.. as you did there. I think I understood it from this example. Yes it is crypto exchanges. I didn't know that they were delayed. I think the library only use HTTP requests but I will check into that. Thanks for that information also! — Andreas, Feb 22 '19 at 22:37

Exchange sync for loop to async approach?

2 Answers2