0

I have a large dataset in the form:

data = [{ a: 12, b: 8 }, { a: 2, c: 4, d: 14 }, { c: 2, e: 4, f: 14 }]

What I want is an object with all keys (here a-f) and the sum of their values across the data set, like so:

{ a: 14, b: 8, c: 6, d: 14, e: 4, f: 14 }

I can get the desired result like this:

function sum(a, b) { return a + b };

function countTotal(n) {
  let ndata = data.filter((i) => Object.keys(i).includes(n))
  let cnt = Object.assign(ndata.map((i) => i[n])).reduce(sum);  
  return {[n]:cnt};
};

let names = 'abcdef'.split('')
let res = Array.from(names).map((n) => countTotal(n))
res = Object.assign({}, ...res);

My problem is that this takes quite long for the actual data set I have (which is quite big). Is there a way to do the same more efficiently?

Below is some code do create a large dummy data set approximating the real data set.

let dummy_names = [];
for (let i = 0; i < 2000; i++) {
    dummy_names.push((Math.random() + 1).toString(36).slice(2,7));
};
dummy_names = [...new Set(dummy_names)];
names = new Set();

function makeResponses() {
  let responses = {};
  let idx = 0;
  for (let j = 0; j <= Math.floor(Math.random() * 7); j++) {
    idx = Math.floor(Math.random()*dummy_names.length);
    inam = dummy_names[idx];
    names.add(inam);
    responses[inam] = Math.floor(Math.random()*20);
  };
  return responses;
};

let data = [];
for (let i = 0; i < 20000; i++) {
  data.push(makeResponses());
    };
Dave Newton
  • 158,873
  • 26
  • 254
  • 302
AndreasM
  • 902
  • 5
  • 10
  • Check out `reduce()`, although it may not be faster. How big is the dataset you're describing? – Rory McCrossan May 10 '23 at 14:16
  • does it really need to be a sinble object with the sum of all keys? or it can be an array of objects with that sum and a single object in it? – Chris G May 10 '23 at 14:18
  • You cannot do that under O(nxk) – EugenSunic May 10 '23 at 14:22
  • 2
    I'm not sure I understand the approach taken in the first snippet; can't you iterate once over the data array, then iterate over the keys in each data object and sum into the result object? Is that what it's doing already? I'm not finding the code easy to reason about *(admittedly on mobile, but still)*. – Dave Newton May 10 '23 at 14:23
  • @DaveNewton No, that is not what I was doing (and should have done what you suggest and the response below does). I got stuck on the notion of iterating over the keys of interest first and looking up the values in the data set based on that. I was filtering the data in each iteration (to objects containing the current key to improve performance a bit) before summing the relevant values. – AndreasM May 10 '23 at 17:27

1 Answers1

4

I'd use a helper object to keep track of the sums and loop through the objects in the array.

The most important thing is only look at each value once to keep the complexity (in terms of O notation) low. There are many ways to iterate, I'm not sure if for loops or .forEach is faster.

Here is a rough solution:

    const data = [{a: 12, b: 8}, {a: 2, c: 4, d: 14}, {c: 2, e: 4, f: 14}];
    const sums = {};
    data.forEach(object => {
        Object.entries(object).forEach(([key, value]) => {
            if (sums.hasOwnProperty(key)) {
                sums[key] += value;
            } else {
                sums[key] = value;
            }
        });
    });
    console.log(sums);
Dave Newton
  • 158,873
  • 26
  • 254
  • 302
timotgl
  • 2,865
  • 1
  • 9
  • 19
  • you copy and pasted that from chatgpt – EugenSunic May 10 '23 at 14:22
  • 3
    @EugenSunic It's pretty basic, I just wrote it in the console of my Chrome browser. It took like a minute. But thanks for accusing me out of nowhere. – timotgl May 10 '23 at 14:23
  • @EugenSunic Why? It's pretty much the approach I would have taken *(see comment)* and it's like two minutes of thinking and typing :shrug: – Dave Newton May 10 '23 at 14:25
  • ChatGPT answers generally include lots of chatty text and a tell-tale summary paragraph at the end. I seriously doubt it could write the above function; not impossible, but I doubt it. – Pointy May 10 '23 at 14:25
  • because you don't use hasOwnProperty for those things, totally unessceary – EugenSunic May 10 '23 at 14:26
  • 1
    Some people use `.hasOwnProperty()` a lot; it seems to be a coding style and I've definitely seen it before. – Pointy May 10 '23 at 14:26
  • @EugenSunic It's not how I would have done it, but it's a perfectly reasonable, generic way to see if a key is present in an object. – Dave Newton May 10 '23 at 14:27
  • 3
    could you please add some details about your answer plz? not questioning whether this was a copy paste from chatGPT or not but an answer with just code and no explanation is still a bad answer – Chris G May 10 '23 at 14:28
  • 1
    @ChrisG Fair point, I added some description. – timotgl May 10 '23 at 14:31
  • For reference to @ChrisG 's comment, see this meta post: https://meta.stackoverflow.com/questions/300837/what-comment-should-i-add-to-code-only-answers – freedomn-m May 10 '23 at 14:34
  • @timotgl Many thanks! Now it seems so obvious to do it like that ... – AndreasM May 10 '23 at 17:19