0

I recently created a bot in node.js, but its super super heavy, uses 100% cpu! I use node-fetch, async.parallel and setInterval, im not sure why it uses 100% cpu, it might be maybe because setInterval queues? Idk. Here is my code though:

async.parallel({
  task1: function () {
    setInterval(function () {
      let itemRandomizer =
        config.generalInfo.itemList[
          Math.floor(Math.random() * config.generalInfo.itemList.length)
        ];
      let cookieRandomizer =
        randomCookies[Math.floor(Math.random() * randomCookies.length)];
      fetch(url, {
        method: "GET",
        agent,
        headers: {
          "Content-Type": "application/json",
          cookie: `cookie=${cookieRandomizer}`,
        },
      })
        .then((res) => res.json())
        .then((json) => {
          checks++;
          //console.log(checks, itemRandomizer, json.data[0].price)
        })
        .catch((err) => {});
    }, 0);
  },
  task2: function () {
    /* same code as task 1 (total 20 tasks) */
  },
});

I do not want to increase the interval, want it to stay as fast as it can, but 100% cpu is unreal, i dont want it to use that much, any work around? Is setInterval cpu-intensive? Threads consume more CPU than async.parallel, so thats why i dont use threads.

Phil
  • 157,677
  • 23
  • 242
  • 245
cooldevnocap
  • 81
  • 5
  • 3
    So you're starting 20 tasks that each start an interval timer with the minimum possible delay between intervals. That's going to be significantly faster than the latency (round-trip time) of your `fetch()` calls. Thus your code is piling up HTTP requests pretty much as fast as it can. – Pointy Apr 11 '23 at 03:33
  • how would I achieve this? What can I do so it does not keep stacking up @Pointy – cooldevnocap Apr 11 '23 at 03:35
  • 1
    Well what exactly is it that you're trying to achieve? – Pointy Apr 11 '23 at 03:36
  • Im trying to not lose speeds, but should use less cpu, im fine with it using less cpu but more threads, but 20 is too little for me – cooldevnocap Apr 11 '23 at 03:37
  • You realize that the `fetch()` calls are asynchronous; the call to `fetch()` returns immediately but the HTTP request finishes when it finishes, at which point the `.then()` callbacks are invoked. The interval function does not wait for that to happen (and it can't). – Pointy Apr 11 '23 at 03:38
  • so how would I prevent this? – cooldevnocap Apr 11 '23 at 03:39
  • 2
    _"Im trying to not lose speeds"_... that does not describe at all what you're trying to do. What is the intention of this application? It looks like you want to continuously poll URLs. Should each poll request wait for the previous to complete? Should anything happen to stop polling? How should errors be dealt with? – Phil Apr 11 '23 at 03:40
  • I hired someone on fiverr to make a website with apis, im checking the durability of those apis as he said it can handle a lot of requests and block bots – cooldevnocap Apr 11 '23 at 03:51

1 Answers1

2

You're using setInterval with a delay of zero, therefore you are scheduling basically infinite tasks to run. That's crazy. What you probably want is to start a new task only after one task is done (or maybe start a few, but at some point wait until at least one task is completed).

By the way, consider using something like p-map so that you can easily configure concurrency to a value like 20 instead of trying to basically re-implement what p-map already does.

Pedro A
  • 3,989
  • 3
  • 32
  • 56