Compare two elemens in Array and if certain properties match, update property of the first array

Question

I have two arrays.

allInventoryItems.length //100,000 objects

`allInventoryItems[0] // {
   name: 'Something',
   price: 200
}

and

currentInventory.length //~250 objects

`currentInventory[0] // {
   name: 'Something Else',
   price: 300
}

You get the point.

What I want to do is compare these two arrays, and if their name property matches, update currentInventory's price property to the price of the allInventoryItems .

What is the best, and fastest way to loop through such a big array of objects, and how can I do it so that I don't have to loop 100,000 times through the allInventoryItems?

This is my current Model:

 updatePrice() {
        let allInventoryItems = [];
        Price.find({})
          .then(items => {
            allInventoryItems.push(items[0].items);
        })

        return new Promise((resolve, reject) => {
            Inventory.find({
            }).then(items => {
                // Update items[0].price according to allInventory items 
                   price
            });
        });
    }

Update: Sorry for not being more specific.

Okay, so my Collections are like this.

Inventory:

Prices:

Now, the Prices Collection is ALL the items, ever. And the Inventory Collection is the items that the user has in his Inventory.

The process is, that I want to update all the prices in the USER's Inventory, with the prices from the Prices Collection, where the Inventory[item].market_hash_name and the Prices[item].market_hash_name match.

Thanks!

What is the role of `game_id` here? Are there several of them? Should they match with something? — trincot, Feb 16 '18 at 20:12
gameid is the Steam's corresponding number for the game (730 = Counter Strike). I currently have 3 Documents in the Prices Collection and they all have a different gameid. Meaning the items in each Document are from a different game. But for this example lets say the query is this: Price.find({ gameid: RequestConfig.appid.DOTA2 }).then... — filipbarak, Feb 16 '18 at 20:17
The inventory is always for that particular game, or should there also be a similar filter on the inventory? — trincot, Feb 16 '18 at 20:21
Well, I am indeed planning to add a property in the items Array of the Inventory Collection named `appid` which will correspond to the id of the game. So the items array for a users inventory will have all kinds of items, for all kinds of games, separated only by the appid property which I will add. — filipbarak, Feb 16 '18 at 20:24
@FilipBarakovski For future reference, editing a question that alters it completely after people have answered the original question makes it very confusing for both the people trying to help and then also anyone who might find this question at some point in the future. If you need to alter the question in such a way it's better to ask a new one. — Jason Cust, Feb 16 '18 at 22:05
Thanks for the heads up @JasonCust. Will remember that for next time — filipbarak, Feb 16 '18 at 22:07

score 1 · Answer 1 · answered Feb 16 '18 at 19:18

1

Rather than allInventoryItems being an array, you could make it an object and use it as a hashtable. Then, lookups would be way faster than finding each item in an array.

 updatePrice() {
        let allInventoryItems = {};
        Price.find({})
          .then(items => {
            // this might be wrong depending on structure of items
            allInventoryItems[items[0].items.name] = items[0].items.price;
        })

        return new Promise((resolve, reject) => {
            Inventory.find({
            }).then(items => {
                // loop through currentItems.  
                // If allInventoryItems[currentItem.name] exists, set the price of currentItem to its value.  This lookup is O(1).
            });
        });
    }

answered Feb 16 '18 at 19:18

James

20,957
5
26
41

Hey! Thanks for the answer I see how that would be doable. I got a small question though, doesn't there need to be a loop after we get the `items` from the Promise? – filipbarak Feb 16 '18 at 19:36
I meant the the 1st promise, where we need to construct the hashtable :) Something like: allInventory => { allInventory[0].items.forEach(item => { allInventoryItems[item.market_hash_name] = item.price; }) – filipbarak Feb 16 '18 at 19:45
What is that index `[0]` about? Is there a `[1]`? – trincot Feb 16 '18 at 20:03
Yes, if `items` is an array - it looked like items[0].items was an individual item from `allInventoryItems.push(items[0].items);` which is where [0] comes from. – James Feb 16 '18 at 20:05
It might be easier to have the backend return a JSON string encoding the complete allInventoryItems hashtable than to loop through it in javascript, if possible. – James Feb 16 '18 at 20:06

trincot · Accepted Answer · 2018-02-16T20:34:31.080

1

As you stored all the items in the same document, I fear there is not much improvement you can bring, as you always have to update the whole document. Still you can use a Map to avoid a nested loop. You'll still have to perform both loops though, but consecutively.

return Promise.all(Inventory.findOne({ appid: RequestConfig.appid.DOTA2 }), 
                   Price.findOne({ gameid: RequestConfig.appid.DOTA2 }))
              .then( ([inv, ref]) => {
    // Create a map, allowing to get an inventory item by name in constant time
    const map = new Map(inv.items.map(item => [item.market_hash_name, item]));
    for (const refItem of ref.items) {
        if (map.has(refItem.market_hash_name) {
            // Update a single price in the inventory document
            map.get(refItem.market_hash_name).price = ref.price;
        }
    )
    return inv.save(); // Update the whole inventory document, and return the promise
);

NB: Note that your code employs the promise construction anti-pattern: don't create a new Promise if you already have one.

edited Feb 16 '18 at 20:34

answered Feb 16 '18 at 19:48

trincot

317,000
35
244
286

Thanks for the answer trincot. I'm still studying mongoose so forgive me if I am asking dumb questions. I see the pattern here. I have 1 question though, my Collection of Inventory looks like this https://prnt.sc/ifr0fm. So Inventory.findOneAndUpdate in that shape would not work. How do you suggest I work around nested properties like these using mongoose? Something like. Inventory.findOneAndUpdate(gameid: gameid, { $set: { // get the nested items corresponding? } }) Thanks for your much appreciated help! – filipbarak Feb 16 '18 at 19:54
Can you update your question to give the actual fields and structure there? It will become confusing if your question is about one structure (name, price), and you speak here about gameid and nested items. – trincot Feb 16 '18 at 20:02
Updated! I hope you can take a look! Thanks. – filipbarak Feb 16 '18 at 20:09
Given the OP's provided example numbers for each collection, 100,000 documents for `Price` and only 250 documents for `Inventory`, this would result in 99,750 no-op `findOneAndUpdate` calls. – Jason Cust Feb 16 '18 at 20:32
I have updated my answer based on the OP's clarifications. – trincot Feb 16 '18 at 20:34
Thanks @trincot. I sincerely appreciate the time you invested to answer my question. Just a small thing, do you think it's a better design if I rewrite my DB collections, so they have 100,000 individual documents in the Price, instead of One Document with 100,000 items in the items array? – filipbarak Feb 16 '18 at 20:48
Yes, I do think so. Then you will be able to create an index on the fields you want to find by, and you'll only have to update the parts that need updating, not the whole collection of items. – trincot Feb 16 '18 at 20:54

Jason Cust · Answer 3 · 2018-02-16T20:38:13.927

In general, since Inventory is likely to be a much smaller list that will be iterated over, then it would be a much smaller loop to do a lookup and update then to loop over the entire Price collection and try to find a possible match in Inventory. For instance given the numbers in your example: 100,000 documents for Price and only 250 documents for Inventory would be a savings of 99,750 no-op loops.

As an alternative if you are using node 8 then you can use async/await with cursors to limit the amount of in memory processing.

async function updatePrices() {
  const inventoryCursor = Inventory.find({}).cursor();
  let item;

  while (item = await inventoryCursor.next()) {
    const price = await Price.findOne({
      name: item.name
    }).exec();
    console.log({
      item,
      price
    });
    item.price = price.price;

    const updated = await item.save();
    console.log({
      updated
    });
  }
}

The only problem here is that my items are actually nested (I updated my question), so I cannot execute this query. Which is why my question is so complicated =/ — filipbarak, Feb 16 '18 at 20:52

Compare two elemens in Array and if certain properties match, update property of the first array

3 Answers3