Test multiple parsing outputs instead of hardcoding each of them into a constant Node js

Question

So in my application I am cron parsing multiple database data and I have it right now to where I create a const run for each data to find the dag_id and test the determineCron one at a time manually. How would I make an array so that the "determineCron()" at the bottom will test every run instead of me just manually coding determineCron(run), determineCron(run2), determineCron(run3), determineCron(run4) etc.

const run = {
    start_date: '2022-07-01T13:45:01.221636+00:00',
        end_date: '2022-07-01T14:11:01.864293+00:00',
    state: 'success',
    dag_run_id: 'scheduled__2022-06-30T13:45:00+00:00',
    dag_id: 'IPP_CYCLE_PARMS',
}

const run2 = {
    start_date: '2022-06-26T23:00:00.742495+00:00',
    end_date: '2022-06-27T14:10:23.108401+00:00',
    state: 'failed',
    dag_run_id: 'scheduled__2022-06-25T23:00:00+00:00',
    dag_id: 'EFS-Winning-Route-daily-batch'
}

async function determineCron(result){
    /*dagID = result?.[0]?.dag_id || 0*/
    dagID = result ? result.dag_id : 0
    console.log(dagID)
    job = await bqConnection().query(`SELECT * FROM \`np-inventory-planning-thd.IPP_SLA.expected_sla\` where dag_id = "${dagID}"`)
    console.log(job[0][0]);
    cronTime = job[0][0].cron_time
    var interval = parser.parseExpression(cronTime);
    console.log('Date: ', interval.next().toString());
}
determineCron(run)
determineCron(run2)

In addition I have the cronTime as job[0][0], if there is a way to make an array to loop through each run that would be good too — Josh Anderson, Jul 08 '22 at 15:36

score 0 · Answer 1 · answered Jul 08 '22 at 15:42

0

const runs = [
{
    start_date: '2022-07-01T13:45:01.221636+00:00',
        end_date: '2022-07-01T14:11:01.864293+00:00',
    state: 'success',
    dag_run_id: 'scheduled__2022-06-30T13:45:00+00:00',
    dag_id: 'IPP_CYCLE_PARMS',
},
{
    start_date: '2022-06-26T23:00:00.742495+00:00',
    end_date: '2022-06-27T14:10:23.108401+00:00',
    state: 'failed',
    dag_run_id: 'scheduled__2022-06-25T23:00:00+00:00',
    dag_id: 'EFS-Winning-Route-daily-batch'
}
]
runs.forEach(determineCron)

answered Jul 08 '22 at 15:42

Konrad

21,590
4
28
64

Hi, so how would i test each of the parsed data in the determineCron function that would just test the runs data – Josh Anderson Jul 08 '22 at 15:50
it is giving me an error throw new Error('Validation error, cannot resolve alias "' + match + '"'); – Josh Anderson Jul 08 '22 at 15:51

JSmart523 · Answer 2 · 2022-07-10T08:33:10.873

0

Put your runs into an array and then use array.forEach, like this:

interface CronRun {
  start_date: string;
  end_date: string;
  state: string;
  dag_run_id: string;
  dag_id: string;
}

const runs: CronRun[] = [
  {
    start_date: '2022-07-01T13:45:01.221636+00:00',
    end_date: '2022-07-01T14:11:01.864293+00:00',
    state: 'success',
    dag_run_id: 'scheduled__2022-06-30T13:45:00+00:00',
    dag_id: 'IPP_CYCLE_PARMS',
  },
  {
    start_date: '2022-06-26T23:00:00.742495+00:00',
    end_date: '2022-06-27T14:10:23.108401+00:00',
    state: 'failed',
    dag_run_id: 'scheduled__2022-06-25T23:00:00+00:00',
    dag_id: 'EFS-Winning-Route-daily-batch'
  }
];

async function determineCron(result: ChronRun){
    /*dagID = result?.[0]?.dag_id || 0*/
    dagID = result ? result.dag_id : 0
    console.log(dagID)
    job = await bqConnection().query(`SELECT * FROM \`np-inventory-planning-thd.IPP_SLA.expected_sla\` where dag_id = "${dagID}"`)
    console.log(job[0][0]);
    cronTime = job[0][0].cron_time
    var interval = parser.parseExpression(cronTime);
    console.log('Date: ', interval.next().toString());
}

runs.forEach(determineCron);

like your original code, this will run them all in parallel. If you want them in serial you'll have to do

for (const run of runs) {
  await determineCron(run);
}

instead.

edited Jul 10 '22 at 08:33

answered Jul 08 '22 at 15:51

JSmart523

2,069
1
7
17

interface CronRun { ^^^^^^^ SyntaxError: Unexpected identifier its giving this error for CronRun – Josh Anderson Jul 08 '22 at 15:56
interfaces should not be inside of classes. Is that the problem? – JSmart523 Jul 08 '22 at 15:58
Yes i believe so its saying thats its an unexpected identifier – Josh Anderson Jul 08 '22 at 15:59
Then move the interface declaration to it's own file and import it. Best practice is to do this anyway. – JSmart523 Jul 08 '22 at 17:36
1

You have to use `for await (..) {...}` https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of – Michael Brenndoerfer Jul 08 '22 at 17:59
@MichaelBrenndoerfer Wait, are you sure? `(async function() { const nums = [500, 100, 2]; const delay = async function (ms) { return new Promise(resolve => setTimeout(resolve, ms)).then(() => console.log(ms));}; for (const ms of nums) { await delay(ms); }})()` prints out 500, 100, and 2 as expected. – JSmart523 Jul 08 '22 at 18:07
Yes, positive. If you want to use await in a for of loop, use the for await version of it – Michael Brenndoerfer Jul 08 '22 at 21:59
@MichaelBrenndoerfer, I played around more, looked more, and `for await` is only best for asynchronous iterators. See [//stackoverflow.com/a/59695815/7158380](https://stackoverflow.com/a/59695815/7158380). – JSmart523 Jul 10 '22 at 08:31
Yes of course, you'd use `for await` only if you have async function calls in your loop. It's the case in above example, see your `await determineCron(run);` If you "fire and forget", then you don't need it, but if you want the iteration of the loop to wait until your asnyc function actually returns something, you'd use `for await` – Michael Brenndoerfer Jul 10 '22 at 21:29
No. It's for asynchronous iterators, not a synchronous iterator that happens to return promises. `for (const ms of nums) { await delay(ms); }; console.log('fin');` prints "fin" last. `for await (const ms of nums) { delay(ms); }; console.log('fin');` does not. For me, anyway. ¯\\_(ツ)_/¯ – JSmart523 Jul 11 '22 at 03:02

score 0 · Accepted Answer · answered Jul 08 '22 at 16:46

0

You could create an array with the references of the constants, but it's tricky after that, because if you use any built-in array method like map, forEach, reduce... to execute the function it will not await the previous finish to execute the next call. Therefore, what you should do is to use for and their respective variations..

// if you are in a async scope you can remove that self invoked function

(async function () {
  const runs = [
    {
      start_date: "2022-07-01T13:45:01.221636+00:00",
      end_date: "2022-07-01T14:11:01.864293+00:00",
      state: "success",
      dag_run_id: "scheduled__2022-06-30T13:45:00+00:00",
      dag_id: "IPP_CYCLE_PARMS",
    },
    {
      start_date: "2022-06-26T23:00:00.742495+00:00",
      end_date: "2022-06-27T14:10:23.108401+00:00",
      state: "failed",
      dag_run_id: "scheduled__2022-06-25T23:00:00+00:00",
      dag_id: "EFS-Winning-Route-daily-batch",
    },
  ];

  async function determineCron(result) {
    /*dagID = result?.[0]?.dag_id || 0*/
    dagID = result ? result.dag_id : 0;
    console.log(dagID);
    job = await bqConnection().query(
      `SELECT * FROM \`np-inventory-planning-thd.IPP_SLA.expected_sla\` where dag_id = "${dagID}"`
    );
    console.log(job[0][0]);
    cronTime = job[0][0].cron_time;
    var interval = parser.parseExpression(cronTime);
    console.log("Date: ", interval.next().toString());
  }

  for (let run of runs) {
    await determineCron(run);
  }
})();

answered Jul 08 '22 at 16:46

Bernardo Moraes

130
8

cronTime = job[0][0].cron_time ^ TypeError: Cannot read properties of undefined (reading 'cron_time') – Josh Anderson Jul 08 '22 at 17:08
So I am getting this error with the job[0][0] do you know by any chance on how i would change this. cron_time is a field in that bigquery table (expected_sla) – Josh Anderson Jul 08 '22 at 17:08
What is the log of console.log(job[0][0])? If you're using the self invoked function, you need to encapsulate all your code inside it. – Bernardo Moraes Jul 08 '22 at 17:46
Thank you for pointing out that array.forEach(await f) doesn't work! I hadn't known that. – JSmart523 Jul 08 '22 at 17:52
{ dag_id: 'oclt-leadtime-daily', cron_time: '00 23 * * 1-6', expected_end_time: '23:10:00' } – Josh Anderson Jul 08 '22 at 18:00
this is an example of what job[0][0] log outputs so I have a total of 12 different data and after the 11th run it outputs an error for the job[0][0] and I am not sure. Is there a better way to do this? – Josh Anderson Jul 08 '22 at 18:04
Never mind it looks like i did not add that to the bigquery thank you guys for the help – Josh Anderson Jul 08 '22 at 18:10

Test multiple parsing outputs instead of hardcoding each of them into a constant Node js

3 Answers3