Looping 2 arrays in JavaScript not working correctly

Question

I've got the following JavaScript code which as far as I know should output the date and the corresponding calculated value only once. However for some reason, I'm seeing incorrect output inside the parseData function with dates logging twice and values getting overwritten. I'm not sure what is causing the issue here: Here's the entire gist that can be run with node.js

var express = require("express")
const NIFTY50_SYMBOL = 'NIFTY 50'
const NIFTYSMALLCAP250_SYMBOL = 'NIFTY SMLCAP 250'
const startDate = '02-Jan-2007'
const requestUrl = `https://www.niftyindices.com/Backpage.aspx/getTotalReturnIndexString`
const requestHeaders = {
  'Content-type': 'application/json; charset=UTF-8',
  'Accept-Language': 'en-GB',
  'Accept': '*/*'
}

const PORT = process.env.PORT || 3000

var app = express()
var path = require('path')
app.use(express.static(path.join(__dirname,"public")))

app.listen(PORT, () => {
  console.log("Server running on port " + PORT)
})

function getTodayAsString() {
  let today = new Date()
  let options = { day: '2-digit', month: 'short', year: 'numeric' }
  return today.toLocaleDateString('en-GB', options).replace(/ /g, '-')
}

async function fetchDataAsync(url, headers, body) {
  try {
    const response = await fetch(url, {
      method: 'POST',
      body: JSON.stringify(body),
      headers: headers,
    })
    const data = await response.json()
    return data
  } catch (error) {
    console.log('Error:', error)
  }
}

app.get("/", (req, res) => {
  const NIFTY50_REQ_BODY = {
    'name': NIFTY50_SYMBOL,
    'startDate': startDate,
    'endDate': '27-Mar-2023'//getTodayAsString()
  }

  const NIFTYSMALLCAP250_REQ_BODY = {
    'name': NIFTYSMALLCAP250_SYMBOL,
    'startDate': startDate,
    'endDate': '27-Mar-2023'//getTodayAsString()
  }

  const promise1 = fetchDataAsync(requestUrl, requestHeaders, NIFTY50_REQ_BODY)
  const promise2 = fetchDataAsync(requestUrl, requestHeaders, NIFTYSMALLCAP250_REQ_BODY)

  Promise.all([promise1, promise2])
  .then(([json1, json2]) => {
    parseData(json1.d, json2.d)
    res.send('Finished parsing data')
  })
  .catch(error => console.log('Error:', error))
})

const smallcapInitialTRI = parseFloat(2100.00)
const niftyInitialTRI = parseFloat(4807.77)

function parseData(niftyData, smallcapData) {
  const linebreak = "=".repeat(50);
  console.log(linebreak)
  console.log("Start parsing data:")
  console.log(linebreak)

  const niftyValues = JSON.parse(niftyData)
  const smallcapValues = JSON.parse(smallcapData)

  for(const smallcapObj of smallcapValues) {
    const smallcapDate = smallcapObj.Date
    const smallcapTRI = parseFloat(smallcapObj.TotalReturnsIndex)
    let niftyTRI = 0

    for (const niftyObj of niftyValues) {
      const niftyDate = niftyObj.Date

      if (niftyDate === smallcapDate) {
        niftyTRI = parseFloat(niftyObj.TotalReturnsIndex)
        console.log("date: "+ smallcapDate + " niftyTRI: " + niftyTRI + " smallcapTRI: " + smallcapTRI)
        break
      }
    }

    if (smallcapTRI && niftyTRI) {
      const relativeValue = ((smallcapTRI / smallcapInitialTRI) / (niftyTRI / niftyInitialTRI))
      console.log("date: "+ smallcapDate + " relative value: " + relativeValue)
    }
  }

  console.log(linebreak)
  console.log("Finished parsing data:")
  console.log(linebreak)
}

module.exports = app

The output is sometimes incorrectly logged twice. I've shared the entire output in the linked gist comments. The JSON response of nifty_50_data & nifty_smallcap_data can be found here & here respectively as the response is way too big to insert into the post.

I tried logging the values inside the if (nifty_smallcap_tri && nifty_50_tri) { block and I'm seeing weird behaviour, there are two passes of the loops happening for the same date. I had thought it would have always been one-pass since I'm trying to match the dates in the inner for loop which ideally should happen only once given that dates occur only once:

Log 1:
date: 26 Mar 2010 nifty_smallcap_initial: 2100 nifty_50_initial: 4807.77
date: 26 Mar 2010 nifty_smallcap_tri: 2824.34 nifty_50_tri: 2824.34
date: 26 Mar 2010 relative value: 2.289414285714286

And

Log 2:
date: 26 Mar 2010 nifty_smallcap_initial: 2100 nifty_50_initial: 4807.77
date: 26 Mar 2010 nifty_smallcap_tri: 6562.47 nifty_50_tri: 2824.34
date: 26 Mar 2010 relative value: 5.3195481307390144

What's weirder is that for the date 26 Mar 2010 the nifty_smallcap_tri & nifty_50_tri values as received in the json response are as follows:

Server JSON response values:
{\"Index Name\":\"Nifty Smlcap 250\",\"Date\":\"26 Mar 2010\",\"TotalReturnsIndex\":\"2824.34\"}
{\"Index Name\":\"Nifty 50\",\"Date\":\"26 Mar 2010\",\"TotalReturnsIndex\":\"6562.47\"}

Why in Log 1 are both nifty_smallcap_tri & nifty_50_tri values that of Nifty Smlcap 250 from the response and in Log 2 why are nifty_smallcap_tri & nifty_50_tri values getting swapped from their actual values in the response?

Edit: I verified the parseData function by using the JSON responses as strings and it works fine. Below is a working example with truncated data wherein I'm calling the function with the JSON response as local strings & it works fine. Cannot post the entire data dump due to character limits. I'm not sure what is causing the issue when parsing is done on data fetched via api.

var result_list = []
var tri_data = {}
const nifty_smallcap_initial = 2100.00
const nifty_50_initial = 4807.77

const niftyJSON = "[{\"IndexName\":\"NIFTY50\",\"Date\":\"27Mar2020\",\"TotalReturnsIndex\":\"12193.62\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"26Mar2020\",\"TotalReturnsIndex\":\"12167.15\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"25Mar2020\",\"TotalReturnsIndex\":\"11710.71\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"24Mar2020\",\"TotalReturnsIndex\":\"10983.15\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"23Mar2020\",\"TotalReturnsIndex\":\"10710.41\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"20Mar2020\",\"TotalReturnsIndex\":\"12298.74\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"19Mar2020\",\"TotalReturnsIndex\":\"11620.92\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"18Mar2020\",\"TotalReturnsIndex\":\"11904.02\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"17Mar2020\",\"TotalReturnsIndex\":\"12604.41\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"16Mar2020\",\"TotalReturnsIndex\":\"12928.18\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"13Mar2020\",\"TotalReturnsIndex\":\"13987.31\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"12Mar2020\",\"TotalReturnsIndex\":\"13474.38\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"11Mar2020\",\"TotalReturnsIndex\":\"14694.31\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"09Mar2020\",\"TotalReturnsIndex\":\"14684.56\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"06Mar2020\",\"TotalReturnsIndex\":\"15440.48\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"05Mar2020\",\"TotalReturnsIndex\":\"15833.21\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"04Mar2020\",\"TotalReturnsIndex\":\"15805.53\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"03Mar2020\",\"TotalReturnsIndex\":\"15878.98\"},{\"IndexName\":\"NIFTY50\",\"Date\":\"02Mar2020\",\"TotalReturnsIndex\":\"15632.14\"}]"

const smallcapJSON = "[{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"27Mar2020\",\"TotalReturnsIndex\":\"3867.88\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"26Mar2020\",\"TotalReturnsIndex\":\"3834.21\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"25Mar2020\",\"TotalReturnsIndex\":\"3705.15\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"24Mar2020\",\"TotalReturnsIndex\":\"3614.88\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"23Mar2020\",\"TotalReturnsIndex\":\"3649.74\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"20Mar2020\",\"TotalReturnsIndex\":\"4164.75\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"19Mar2020\",\"TotalReturnsIndex\":\"4073.10\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"18Mar2020\",\"TotalReturnsIndex\":\"4293.66\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"17Mar2020\",\"TotalReturnsIndex\":\"4570.10\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"16Mar2020\",\"TotalReturnsIndex\":\"4677.61\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"13Mar2020\",\"TotalReturnsIndex\":\"4945.46\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"12Mar2020\",\"TotalReturnsIndex\":\"4911.61\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"11Mar2020\",\"TotalReturnsIndex\":\"5386.57\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"09Mar2020\",\"TotalReturnsIndex\":\"5400.60\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"06Mar2020\",\"TotalReturnsIndex\":\"5663.56\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"05Mar2020\",\"TotalReturnsIndex\":\"5795.22\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"04Mar2020\",\"TotalReturnsIndex\":\"5768.55\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"03Mar2020\",\"TotalReturnsIndex\":\"5859.40\"},{\"IndexName\":\"NiftySmlcap250\",\"Date\":\"02Mar2020\",\"TotalReturnsIndex\":\"5790.35\"}]"

function parseData(nifty_50_data, nifty_smallcap_data) {
  console.log("==============================================================")
  console.log("Start parsing data:")
  console.log("==============================================================")

  const niftySmallcapValues = JSON.parse(nifty_smallcap_data)
  const niftyFiftyValues = JSON.parse(nifty_50_data)

  for(const nifty_smallcap of niftySmallcapValues) {

    const date = nifty_smallcap.Date
    const nifty_smallcap_tri = parseFloat(nifty_smallcap.TotalReturnsIndex)
    let nifty_50_tri = 0

    for (const nifty_50 of niftyFiftyValues) {
      if (nifty_50.Date === nifty_smallcap.Date) {
        nifty_50_tri = parseFloat(nifty_50.TotalReturnsIndex)
        break
      }
    }

    if (nifty_smallcap_tri && nifty_50_tri) {
      const relative_value = ((nifty_smallcap_tri / nifty_smallcap_initial) / (nifty_50_tri / nifty_50_initial))
      console.log("date: "+ date + ", relative value: " + relative_value)

      tri_data['date'] = nifty_smallcap.Date
      tri_data['nifty_smallcap_tri'] = nifty_smallcap_tri
      tri_data['nifty_50_tri'] = nifty_50_tri
      tri_data['relative_value'] = relative_value

      result_list.push(tri_data)
    }
  }

  console.log("==============================================================")
  console.log("Finished parsing data:")
  console.log("==============================================================")
}

parseData(niftyJSON, smallcapJSON)

@NickParsons, sorry about that. I've fixed it now. Please check. — batman, Mar 28 '23 at 12:30
Please include *samples* of the data included; enough so that we can reproduce the issue. See [mre] for more. — Heretic Monkey, Mar 29 '23 at 03:53
Hi @HereticMonkey, i've provide the entire gist which is linked in the question. It can be run with node.js. Is that not enough? I'm new to JS so I'm not sure i can produce an example that replicates the same behaviour. — batman, Mar 29 '23 at 06:29
@HereticMonkey, I've added a working example which works with the data parsed as local json strings. The issue does not occur even with the entire data of several years if used as local JSON. I'm doing something wrong when parsing the date from the API. — batman, Mar 30 '23 at 12:51

Dimava · Accepted Answer · 2023-04-02T16:05:42.657

1

Your problem lies in the niftyindices.com

If you try to log it as

async function count() {
    const promise1 = fetchDataAsync(requestUrl, requestHeaders, NIFTY50_REQ_BODY);
    const promise2 = fetchDataAsync(requestUrl, requestHeaders, NIFTYSMALLCAP250_REQ_BODY, );
  const [json1,json2] = await Promise.all([promise1, promise2])
    
    const [niftyValues,smallcapValues] = [json1, json2].map(j=>JSON.parse(j.d))
    console.log({
        nifty: niftyValues.length,
        small: smallcapValues.length
    })
}
for (let i = 0; i < 30; i++) {
    await count()
}

you'll see that it's result is pretty random

{ nifty: 4023, small: 4023 }
{ nifty: 5814, small: 8047 }
{ nifty: 8046, small: 12069 }
{ nifty: 4023, small: 12069 }
{ nifty: 12069, small: 9325 }
{ nifty: 12069, small: 8046 }
{ nifty: 4023, small: 8046 }
{ nifty: 8046, small: 4023 }
{ nifty: 4023, small: 8046 }
{ nifty: 4023, small: 8046 }
{ nifty: 4023, small: 8046 }
{ nifty: 4023, small: 8046 }
{ nifty: 12069, small: 8046 }
{ nifty: 8046, small: 4023 }
{ nifty: 8046, small: 4023 }
{ nifty: 4023, small: 8046 }
{ nifty: 8047, small: 12070 }
{ nifty: 4023, small: 6454 }

So, Don't Promise.all, await each of them in order

    const json1 = await fetchDataAsync(requestUrl, requestHeaders, NIFTY50_REQ_BODY);
    const json2 = await fetchDataAsync(requestUrl, requestHeaders, NIFTYSMALLCAP250_REQ_BODY);
    parseData(json1, json2)

and check if the array length is correct (or at least the same for both arrays)

edited Apr 02 '23 at 16:05

answered Mar 31 '23 at 12:10

Dimava

7,654
1
9
24

Note: I've got 12069 (x3) when running a parallel test in Chrome, so this bug is real – Dimava Mar 31 '23 at 12:14
I had not considered that it could be an issue with the response. Might explain why sometimes I'd get the correct result. I will try your suggestions in a few hours. It would be great if you could update your answer with how to async await 2 api calls without `Promise.all` and run some code on completion of both. – batman Mar 31 '23 at 16:10
@batman, to run multiple async calls and run some code on completion of both, you can try to do this as well: `const fetch1 = fetchDataAsync(...); const fetch2 = fetchDataAsync(...); const json1 = await fetch1; const json2 = await fetch2;` See https://stackoverflow.com/a/45286517 for more details – Lucas P. Luiz Apr 02 '23 at 15:47
@batman I've literally wrote that below "don't promise.all" – Dimava Apr 02 '23 at 16:04
@batman here's your third line so you understand better – Dimava Apr 02 '23 at 16:06
1

@LucasP.Luiz that's the same as Promise.all. The point it, you shouldn't make *two fetch requests* at once or server will respont with bad data. First request should have finished (i.e. awaited) befor the second one starts at all – Dimava Apr 02 '23 at 16:07
Thank you @Dimava, waiting until each request completed resolved the issue. – batman Apr 04 '23 at 04:08

Looping 2 arrays in JavaScript not working correctly

1 Answers1

Linked