Determine the average value across a range of data

Question

I have collected an array of weather data that looks like this:

const data = [{
    "city_name": "London",
    "lat": 51.507351,
    "lon": -0.127758,
    "main": {
      "temp": 289.89,
      "temp_min": 288.468,
      "temp_max": 291.15,
      "feels_like": 287.15,
      "pressure": 1004,
      "humidity": 77
    },
    "wind": { "speed": 5.1, "deg": 230 },
    "clouds": { "all": 90 },
    "weather": [
      {
        "id": 804,
        "main": "Clouds",
        "description": "overcast clouds",
        "icon": "04n"
      }
    ],
    "dt": 1593561600,
    "dt_iso": "2020-07-01 00:00:00 +0000 UTC",
    "timezone": 3600
  },
  ...
];

This data continues in ascending date order (hour by hour), for the last 40 years.
(sample: https://pastebin.com/ciHJGhnq ) - the entire dataset is over 140MB.

From this data, I'd like to obtain the average temperature (object.main.temp) for each Month and Week of month, across the entire dataset.

The question I am trying to answer with my data is:

What is the average temperature for January, across the last 40 years.
What is the average temperature for February, across the last 40 years.
...

(get the temperature of each week in January and divide by the number of weeks, repeat for all of the other Januaries in the dataset and average that out too).

Repeat for remaining months.

The output I am aiming to create after parsing the data is:

{ 
   [ 
     "JANUARY": {
       "weekNumber": {
          "avgWeekTemp": 100.00
       }
       "avgMonthTemp": 69.00,
       ...
     },
     ...
   ]
}

The city name & structure of the objects are always the same, in this case London.

// build a unique number of months
// work through our data to work out the week numbers
// work through the data once again and place the data in the right week inside finalOutput
// work through the final output to determine the average values

Unfortunately I'm not very proficient in JavaScript, so I couldn't get past the second obstacle:

"use strict";

const moment = require("moment");
const data = require("./data.json");

let months = [
  {
    January: [],
  },
  {
    February: [],
  },
  {
    March: [],
  },
  {
    April: [],
  },
  {
    May: [],
  },
  {
    June: [],
  },
  { July: [] },
  { August: [] },
  { September: [] },
  { October: [] },
  { November: [] },
  { December: [] },
];
const finalOutput = [];

finalOutput.push(...months);

data.forEach((object) =>
  finalOutput.forEach((month) => {
    if (
      Object.keys(month)[0] === moment(new Date(object.dt_iso)).format("MMMM")
    ) {
      [month].push(object.dt_iso);
    }
  })
);

console.log(finalOutput);

Which only returned the array of months with nothing in each month.

[
  { January: [] },  
  { February: [] },
  { March: [] },
  { April: [] },
  { May: [] },
  { June: [] },
  { July: [] },
  { August: [] },
  { September: [] },
  { October: [] },
  { November: [] },
  { December: [] }
]

How can I calculate the average values per week & month across my entire data set?

Answer 1

I'm going to write your script for you, but while you wait here's some high-level guidance.

First, let's study your data. Each row is an hourly weather measurement. As a result, each datapoint you want will be an aggregate over a set of these rows. We should organize the script along those lines:

1. We'll write a function that accepts a bunch of rows and returns the arithmetic mean of the temperatures of those rows: function getAvgTemp(rows) => Number

2. We'll write another function that takes a bunch of rows, plus the desired month, and returns all the rows for just that month: function getRowsByMonth(month) => Array(rows)

3. We'll write another function that takes a bunch of rows, plus the desired week number, and returns all the rows for just that week: function getRowsByWeekNumber(rows, weekNumber) => Array(rows)

^^ that's if "week number" means 1-52. But if "week number" means "week within the month," then instead we'll do:

3. A function will also take a month: function getRowsByMonthWeek(rows, month, weekNumber) => Array(rows)

From these basic building blocks, we can write a routine that assembles the data you want.

---

What would that routine look like?

Probably something like this:

1. Loop through all the months of the year. We won't look in the data for these months, we'll hard-code them.
   1. For each month, we'll call getRowsByMonth on the full data set. Call this set monthRows.
   2. We'll pass monthRows to getAvgTemp -- it doesn't care what the timespan is, it just extracts and crunches the temp data it receives. That's our avgMonthTemp solved for.
   3. Depending on what you mean by "week number," we'll divide the monthRows into smaller sets and then pass each set into getAvgTemp. (The hardest part of your script will be this division logic, but that's not to say it will be that hard.) This gives us your weekly averages.
   4. We'll assemble these values into a data structure and insert it into the final structure that ultimately gets returned/logged.

---

Here's the implementation. It's a little different than I expected.

The biggest change is that I did some pre-processing up front so that the date values don't have to be parsed multiple times. While doing that, I also calculate each row's weekNumber. As a consequence, the week logic took the form of grouping rows by their weekNumbers rather than querying the dataset by weekNumber.

Some notes:

• I decided that "weekNumber" means "week-of-year."

Instead of using Moment, I found a week-number algorithm on StackOverflow. If you want to use Moment's algo instead, go ahead.

• The output data structure is not what you described.

Your example is not valid JSON, so I made a guess as to what you had in mind.

Here's an example of what it looks like:

{
  "JUNE": {
    "avgMonthTemp": 289.9727083333334,
    "avgWeekTemps": {
      "25": 289.99106382978727,
      "26": 289.11
    }
  },
  "JULY": {
    "avgMonthTemp": 289.9727083333334,
    "avgWeekTemps": {
      "27": 289.99106382978727,
      "30": 289.11
    }
  }
}

The output will include a top-level entry for every month, whether or not there is any data for that month. However, the avgWeekTemps hash will only have entries for weeks that are present in the data. Both behaviors can be changed, of course.

• It's a reusable script that processes arbitrary JSON files in the format you shared.

You mentioned that each file has data from one city, so I figured you'll be running this on multiple files. I set it up so you can pass the path to the data file as a command-line argument. Note that the CLI logic is not sophisticated, so if you're doing funky things you will have a bad time. Doing CLI stuff well is a whole separate topic.

If your data for London is in a file named london.json, this is how you would process that file and save the results to the file london-temps.json:

$ node meantemp.js london.json > london-temps.json

// meantemp.js

const FS = require('fs')


// sets the language used for month names
// for language choices, see: http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
const MONTH_NAME_LANG_CODE = 'en-US'


// generate the list of month names once
const MONTH_NAMES = Array(12).fill().map(
    ( _, monthNum ) => new Date(2020, monthNum).toLocaleDateString(MONTH_NAME_LANG_CODE, { month: 'long' }).toUpperCase()
)


main()


function main() {
    let filepath = process.argv[2]

    let cityData = readJsonFile(filepath)

    // before working on the data, prep the date values for processing
    let allRows = cityData.map(row => {
        let _date = new Date(row.dt_iso)
        let _weekNum = getWeekNum(_date)
        return { ...row, _date, _weekNum }
    })

    let output = MONTH_NAMES.reduce(( hash, monthName, monthNum ) => {
        // grab this month's rows
        let monthRows = getRowsForMonth(allRows, monthNum)

        // calculate monthly average
        let avgMonthTemp = getMeanTemp(monthRows)

        // calculate weekly averages
        let rowsByWeekNum = groupRowsByWeekNum(monthRows)
        let avgWeekTemps = Object.keys(rowsByWeekNum)
        .reduce(( hash, weekNum ) => ({
            ...hash,
            [weekNum]: getMeanTemp(rowsByWeekNum[weekNum])
        }), {})

        return {
            ...hash,
            [monthName]: { avgMonthTemp, avgWeekTemps }
        }
    }, {})

    console.log(JSON.stringify(output))
}


function readJsonFile( path ) {
    try {
        let text = FS.readFileSync(path, 'utf8')
        return JSON.parse(text)

    } catch ( error ) {
        if(error.code === 'ENOENT') {
            console.error(`Could not find or read path ${JSON.stringify(path)}`)
            process.exit()

        } else if(error instanceof SyntaxError) {
            console.error(`File is not valid JSON`)
            process.exit()

        } else {
            throw error
        }
    }
}


function getRowsForMonth( rows, monthNum ) {
    return rows.filter(row => monthNum === row._date.getUTCMonth())
}


function groupRowsByWeekNum( rows ) {
    return rows.reduce(( hash, row ) => {
        if(!hash.hasOwnProperty(row._weekNum)) {
            hash[row._weekNum] = []
        }

        hash[row._weekNum].push(row)

        return hash
    }, {})
}


// ISO8601-compliant week-of-year function
// taken from https://stackoverflow.com/a/39502645/814463
// modified by me to prevent mutation of args
function getWeekNum( date ) {
    // if date is a valid date, create a copy of it to prevent mutation
    date = date instanceof Date
    ? new Date(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate())
    : new Date()

    let nDay = (date.getDay() + 6) % 7
    date.setDate(date.getDate() - nDay + 3)
    let n1stThursday = date.valueOf()

    date.setMonth(0, 1)

    if (date.getDay() !== 4) {
        date.setMonth(0, 1 + ((4 - date.getDay()) + 7) % 7)
    }

    return 1 + Math.ceil((n1stThursday - date) / 604800000)
}


function getMeanTemp( hourlyReadings ) {
    let temps = hourlyReadings.map(reading => reading.main.temp)
    let mean = getMean(temps)
    return mean
}


function getMean( numbers ) {
    let sum = numbers.reduce(( sum, num ) => sum + num, 0)
    let mean = sum / numbers.length
    return mean
}