Why is this iteration so slow?

Question

I can't seem to figure why this iteration is so slow so I'm hoping someone can help me. This function is used to schedule some actions at certain times, and is looping the list of bots (which will execute the actions) and the list of groups (target of the actions) It takes about 10 minutes to complete with 1300 groups and 50 bots.. Thank you in advance, this is very important!

for (let groupIndex = 0; groupIndex < groups.length; groupIndex++) {
    let group = groups[groupIndex];

    if (!answers.find(a => a.GroupId == group.Id))
        continue;

    for (let botIndex = 0; botIndex < bots.length; botIndex++) {
        var bot = bots[botIndex];
        if (bot.Disabled)
            continue;

        var bot_joined_groups = bots_joined_groups.find(j => j.Id == bot.Id);
        if (bot_joined_groups) {
            bot_joined_groups = bot_joined_groups.Groups;
            if (bot_joined_groups.find(j => j.GroupId == group.Id))
                continue;
        } else
            bot_joined_groups = [];

        var day = 60 * 60 * 24 * 1000;

        var today = new Date();
        if (today < (new Date(bot.WarmingUpEndAt)))
            today = new Date(bot.WarmingUpEndAt);
        else if (Math.floor((Date.parse(today) - Date.parse(bot.LimitAt)) / 86400000) < 7) {
            today = new Date(bot.LimitAt);
            today.setTime(today.getTime() + day * 7);
        }

        var scheduled_joins = scheduled_joins_total.filter(j => j.BotId == bot.Id);

        var schedule_date = new Date();

        if (scheduled_joins.length) {
            schedule_date = scheduled_joins.find(j => {
                let at = new Date(j.At);
                let count = scheduled_joins.filter(c => at.getDate() === new Date(c.At).getDate() && at.getMonth() === new Date(c.At).getMonth() && at.getFullYear() === new Date(c.At).getFullYear()).length;

                if (count < 10)
                    return true;
            });

            if (schedule_date)
                schedule_date = new Date(schedule_date.At);
            else {
                schedule_date = new Date(scheduled_joins[scheduled_joins.length - 1].At);
                schedule_date = new Date(schedule_date.getTime() + day);
            }
        } else
            schedule_date = today;

        let min_hours = 0;
        let min_minutes = 0;

        if (today.getDate() == schedule_date.getDate() && today.getMonth() == schedule_date.getMonth() && today.getFullYear() == schedule_date.getFullYear()) {
            var joined_today_count = bot_joined_groups.filter(c => today.getDate() === new Date(c.JoinedAt).getDate() && today.getMonth() === new Date(c.JoinedAt).getMonth() && today.getFullYear() === new Date(c.JoinedAt).getFullYear()).length;

            if (joined_today_count >= 10) {
                schedule_date = new Date(scheduled_joins_total.length ? (new Date(scheduled_joins_total[scheduled_joins_total.length - 1].At)).getTime() : today.getTime() + day);
                min_hours = 0;
                min_minutes = 0;
            }
            else {
                if (today.getHours() == 23) {
                    schedule_date = new Date(scheduled_joins_total.length ? (new Date(scheduled_joins_total[scheduled_joins_total.length - 1].At)).getTime() : today.getTime() + day);
                    min_hours = 0;
                    min_minutes = 0;
                }
                else {
                    if (schedule_date.getHours() <= today.getHours())
                        min_hours = today.getHours() + 1;

                    joined_today_count++;
                }
            }
        }

        schedule_date.setHours(Math.random() * (23 - min_hours + 1) + min_hours);
        schedule_date.setMinutes(Math.random() * 59 + min_minutes);
        schedule_date.setSeconds(Math.random() * 59);

        var today_joins_count = 0;
        do {
            today_joins_count = scheduled_joins_total.filter(c => schedule_date.getDate() === new Date(c.At).getDate() && schedule_date.getMonth() === new Date(c.At).getMonth() && schedule_date.getFullYear() === new Date(c.At).getFullYear() && c.GroupId === group.Id).length;
            if (today_joins_count)
                schedule_date.setTime(schedule_date.getTime() + day);
        } while (today_joins_count);

        scheduled_joins_total.push({
            BotId: bot.Id,
            GroupId: group.Id,
            At: schedule_date
        });

        values.push(`(${bot.Id}, ${group.Id}, to_timestamp(${schedule_date.getTime() / 1000.0}))`);
    }
}

Really, you should be putting in a timer and console.logging benchmarks to find the lines that are slowest, and the researching alternate approaches. For starters, take a look at https://nikitahl.com/how-to-find-an-item-in-a-javascript-array/ — Kinglish, Nov 29 '21 at 17:51
If you can use an object or `Map` to look up ids instead of calling `find` so many times, the performance should increase considerably. — Unmitigated, Nov 29 '21 at 17:52
Lots of one-off declarations happening in every loop. There are many opportunities for improvements. — Randy Casburn, Nov 29 '21 at 17:58

score 0 · Answer 1 · answered Nov 29 '21 at 18:06

O(n^2)? If

!answers.find(a => a.GroupId == group.Id)

is an array or list with a similar length as groups, then you can expect each iteration to look for 1300 items in memory instead of doing about 1 iterations work. Overall thats 1300 * 1300 times looking values up in memory.

As noted in the comments you can use hash tables or direct lookup tables instead.

Basically try storing the data in a way that lets you retrieve it fast if you're retrieving it often.

In this case you may want a relational database or if you're ok with temporary inconsistency in data you can store the group members inside the original search too, that way you just do one lookup.

Related: https://stackoverflow.com/a/487278/7818349

Why is this iteration so slow?

1 Answers1