0

I have an array consisting of 40,000 company names.

That is, there is such a link on the site https://numfin.com/uk/AAPL/overview

in my case AAPL, this is the name of the company.

I decided to accomplish this task by simply writing these 40k company names into an array, and in a loop just change the company name in the link

I wrote a small script on node js in which I made a loop and changed the names of the companies

import request from "request";
import fs from "fs";
import { tickets } from "./tickets.js"; // array of company names

var data = [];

var out = fs.createWriteStream("tickets.txt", { flags: "a" });

for (let i = 0; i < tickets.length; i++) {
  request(
    `https://numfin.com/uk/${tickets[i]}/overview`,
    function (error, response, body) {
      if (error) {
        console.log("Err: " + error);
        return false;
      }

      if (
        response.statusCode == 200 ||
        response.statusCode == 201 ||
        response.statusCode == 202
      ) {
        console.log(
          `https://numfin.com/uk/${tickets[i]}/overview` + " is alive!!"
        );

        return false;
      }

      if (response.statusCode == 301 || response.statusCode == 302) {
        console.log(
          `https://numfin.com/uk/${tickets[i]}/overview` +
            " is redirecting us!!"
        );
        return false;
      }

      if (response.statusCode == 401) {
        console.log(
          "you are unauthorized to " +
            `https://numfin.com/uk/${tickets[i]}/overview`
        );
        return false;
      } else {
        // IF TICKET === 500 status code, I will write this ticket to file
        console.log(
          `https://numfin.com/uk/${tickets[i]}/overview` + " is dead!!"
        );
        data.push(tickets[i]);
        console.log("data length", data.length);
        out.write(tickets[i].toString());
      }
    }
  );
}

out.end();

I wrote a small script on node js in which I made a loop and changed the names of the companies.

But the problem is that this code does not work and does not write to the file.

And for some reason, when I stuff such a large volume of data, I get the following error:
Error: read ECONNRESET

I'm already tired of trying to solve it, tell me maybe some other options

Vadym Chornyi
  • 23
  • 4
  • Your loop attempts to start every single `request()` function at the same time. Those are asynchronous so the loop does not wait for the `request()` operation to complete before advancing to the next iteration of the loop. Thus, you are probably overwhelming the target host with too many requests or exhausting your own TCP resources. Either way, you should perhaps be doing 10 requests at a time, not 40,000. – jfriend00 Jan 04 '23 at 05:59
  • To see how to run N requests at a time, not all of them, see `mapConcurrent()` [here](https://stackoverflow.com/questions/46654265/promise-all-consumes-all-my-ram/46654592#46654592). And, I'd suggest you switch to using something like `got()` or `node-fetch()` that is promise based instead of the now-deprecated `request()` library that is plain callback-based and no longer being developed. – jfriend00 Jan 04 '23 at 06:01

0 Answers0