0

I wrote a very simple typescript program, which does the following:

  1. Transform users.csv into an array
  2. For each element/user issue an API call to create that user on a 3rd party platform
  3. Print any errors

The excel file has >160,000 rows and there is no way to create them all in one API call, so I wrote this program to run in the background of my computer for ~>20 hours.

The first time I ran this, the code stopped mid for loop without an exception or anything. So, I deleted the user rows from the csv file that were already uploaded and re-ran the code. Unfortunately, this kept happening.

Interestingly, the code has stopped at non-deterministic iterations, one time it was at i=812, another at i=27650, and so on.

This is the code:

const main = async () => {
  const usersFile = await fsPromises.readFile("./users.csv", { encoding: "utf-8" });
  const usersArr = makeArray(usersFile);

  for (let i = 0; i < usersArr.length; i++) {
    const [ userId, email ] = usersArr[i];
    console.log(`uploading ${userId}. ${i}/${usersArr.length}`);
    try {
      await axios.post(/* create user */);
      await sleep(150);
    } catch (err) {
      console.error(`Error uploading ${userId} -`, err.message);
    }
  }
};

main();

I should mention that exceptions are within the for-loop because many rows will fail to upload with a 400 error code. As such, I've preferred to have the code run non-stop and print any errors onto a file, so that I could later re-run it for the users that failed to upload. Otherwise I would have to check whether it halted because of an error every 10 minutes.

Why does this happen? and What can I do?

I run after compiling as: node build/index.js 2>>errors.txt

EDIT: There is no code after main() and no code outside the try ... catch block within the loop. errors.txt only contains 400 errors. Even if it contained another run-time exception, it seems to me that this wouldn't/shouldn't halt execution, because it would execute catch and move on to the next iteration.

Dan
  • 641
  • 1
  • 7
  • 17
  • 1
    And is there something in errors.txt? – derpirscher Jan 17 '21 at 15:10
  • Is there any additional code in the loop's body outside the `try ... catch` block which could throw an exception? You could try `main().catch(e => console.error(e))` which will print any error `main` throws ... Or is there any additional code after `main` which could throw an error? Be aware, that because `main` is async, any code located after the function call to `main()` will immediately be executed. – derpirscher Jan 17 '21 at 15:19
  • Nothing in `errors.txt` besides `400` errors and there's no code outside the `try ... catch`, as well as no code after `main`. Sorry for not mentioning this clearly in the question. Will edit it now. – Dan Jan 17 '21 at 15:38
  • The only other thing I can imagine, is that `makeArray` has an error, and some elements in the `usersArr` are not arrays. So `const [ userId, email ] = usersArr[i];` would throw an error. Have you tried looking at the respective lines in the file (and some lines before or after) And as this assignment is outside of `try catch` it will break out of your `main`. – derpirscher Jan 17 '21 at 15:43
  • Actually `makeArray` was an abstraction for this: `usersFile.split("\n").map((row) => row.split(",")).filter(([userId]) => userId))`. I showed it as a function to try not to distract from the problem. I don't think this could be the problem since the excel file is consistenly formatted. Even if this was the case, then I think I wouldn't have been able to re-run the code after it stopped at a non-formatted row. @derpirscher – Dan Jan 17 '21 at 15:53
  • 1
    It was just an idea, because from the code above, the only possibility I see, that the code could end prematurely in the middle of the loop is 1) a bug in the runtime (or the os) 2) `main` throwing an exception. And the only place for an uncaught exception inside the loop is the assignment of `const [userid, email] = ....` Did you try adding a `main().catch(...)` or placing it inside a `try { main(); } catch(e) {console.error(e);}` block? – derpirscher Jan 17 '21 at 16:04
  • It was a valid idea! you make good points. I'll try this now. Thank you! @derpirscher – Dan Jan 17 '21 at 20:01
  • How is `sleep` implemented? Do you even need to call `sleep`? – Soc Jan 26 '21 at 21:56
  • Like this: `await new Promise((resolve) => setTimeout(resolve, 150))` @Soc – Dan Jan 26 '21 at 22:16

1 Answers1

0

I think this may have been related to this post. The file I was reading was extremely large as noted, and it was saved into a runtime variable. Undeterministically, the OS could have decided that the memory demanded was too high. This is probably a situation to use a Readable Stream instead of a readFile.

Dan
  • 641
  • 1
  • 7
  • 17