Javascript - Array of values linked to a larger array

Question

I am working in Javascript and creating an offline application.

I have a very large array, over 10mb. I need to save elements from the array where a user has made a change, but writing / reading the array to disk is too slow.

I would like to create a second array that holds only the changes made by the user, so I can save that smaller array and then when I need to load changes it can mirror its changes to the large array.

For example:

lib[x][1][y][6] = "no";
libVars[x][1][y][6] = "no";

libVars would simply hold the element changes in lib and would not be the same size as the large array lib as the user will only interact with a small portion of the large array.

Obviously this doesn't work, as libVars doesn't have the same structure as lib, and if it did it would also take up a lot of memory (i think so anyways!)

I would then like to loop through libVars and update lib at the same element points where the changes have been saved in libVars.

Is there a way to save some type of pointer to a location within lib that I could store in libVars with the associated element value?

Any help would be much appreciated.

Thanks.

======================

Sample of my large array

var lib = [
[241,[[221119,"sample data","sample","no","no",131,"no"],
[221121,"sample2 data","sample2","no","no",146,"no"],
[221123,"sample3 data","sample3","no","no",28,"no"],
[221626,"sample4 data","sample4","no","no",26,"no"],
[221628,"sample5 data","sample5","no","no",88,"no"]]],
[330,[[305410,"2sample data","sample 2b","no","no",197,"no"],
[305412,"2sample 2 data","sample2 2b","no","no",147,"no"],
[305414,"3sample 2 data","sample3 2b","no","no",10,"no"] ... 

Attempt at solution posted by Z-Bone

I have added the tempLib var, but I don't see how you eventually update the actual lib object

Object.entries(libVars).forEach(([path, value]) => {

  var tempLib = lib;
  var pathParts = path.split('#');

  pathParts.forEach((pathPart, index) => {

  if (index == pathParts.length-1) {

    tempLib[pathPart] = value;

  } else {
    //I don't fully track what you are doing here. 
    //I assume you are building the array path, but to me it looks like
    // it's getting wiped out and not built upon?    
    tempLib = tempLib [pathPart];
  }
});
});

Answer 1

Update 28/01/2019 Based on nomaam's test data

@nomaam Please check the following snippet. I made it change an arbitrary path in lib array. The path I chose is this: lib[0][1][3][4]. You can see how it initially returns the value "no" and then after execution of the modification tracking solution, it returns the value "yes".

// First Step: Original Data (lib)
var lib = [
[241,[[221119,"sample data","sample","no","no",131,"no"],
[221121,"sample2 data","sample2","no","no",146,"no"],
[221123,"sample3 data","sample3","no","no",28,"no"],
[221626,"sample4 data","sample4","no","no",26,"no"],
[221628,"sample5 data","sample5","no","no",88,"no"]]],
[330,[[305410,"2sample data","sample 2b","no","no",197,"no"],
[305412,"2sample 2 data","sample2 2b","no","no",147,"no"],
[305414,"3sample 2 data","sample3 2b","no","no",10,"no"]]]]

// Log initial value in arbitrary selection lib[0][1][3][4]
console.log(lib[0][1][3][4])

// Second Step: Pointer's object
var libVars = {}

// Example of changing a value via an artificial "pointer" to lib[0][1][3][4]
libVars['0#1#3#4'] = 'yes'; // original value is "no"

// Third Step: Run the modifier - update changes from libVars to lib
Object.entries(libVars).forEach(([path, value]) => {

  var tempLib = lib;
  var pathParts = path.split('#');
  
  pathParts.forEach((pathPart, index) => {
    
    if (index == pathParts.length - 1) {
      
      tempLib[pathPart] = value;
      
    } else {
      
      tempLib = tempLib[pathPart];
      
    }
  });
});


// Log the change fro lib
console.log(lib[0][1][3][4])

---

Original Answer

If I understand your challenge correctly, you could technically hold an object with the path to the property you want to modify as the key and the modified value as its value.

let's say your original object looks like this:

var lib = {a: {b: { c: { d: "yes"}}}}

You could keep a log of changes in a backup object (e.g. value changes from "yes" to "no"). It could look something like this, with . marking a nested property.

var libVars = {};
libVars['a.b.c.d'] = "no";

Then when you would like to update the original large array/object of values, you could do this:

Object.entries(libVars).forEach(([path, value]) => {
    var tempLib = lib;
    // split by nesting indicator -> dot
    var pathParts = path.split('.');
    // iterate all parts of the path to the modified value
    pathParts.forEach((pathPart, index) => {
        if (index == pathParts.length-1) {
            // once you made your way to last index, update value
            tempLib[pathPart] = value;
        } else {
            tempLib = tempLib[pathPart];
        }
    })
})

console.log(lib); // outputs => {a: {b:{c: {d : "no"}}}}

Answer 2

Update 2

Updated my answer, to get rid of lodash, and it appeared to be equal to Z-Bone's answer. His variant is even more clear than mine.

Updated @ 28.01.2019

var lib = [
[241,[[221119,"sample data","sample","no","no",131,"no"],
[221121,"sample2 data","sample2","no","no",146,"no"],
[221123,"sample3 data","sample3","no","no",28,"no"],
[221626,"sample4 data","sample4","no","no",26,"no"],
[221628,"sample5 data","sample5","no","no",88,"no"]]],
[330,[[305410,"2sample data","sample 2b","no","no",197,"no"],
[305412,"2sample 2 data","sample2 2b","no","no",147,"no"],
[305414,"3sample 2 data","sample3 2b","no","no",10,"no"]]]];

// object to track edits
var edits = {
  '0.1.2.2': 'edited'
};

// sample/simple implementation of lodash's 'set' function
function applyEdit(arr, path, value) {
  var items = path.split('.');
  var last = items.length - 1;
  var current = arr;
  items.forEach(function (item, i) {
    if (i !== last) {
      current = current[item];
    }
  });
  current[items[last]] = value;
}

// our restoration function
function patch(arr, edits) {
  Object.keys(edits).forEach(function(key) {
    var newValue = edits[key];
    applyEdit(arr, key, newValue);
  });
}

console.log(lib[0][1][2][2]) // "sample3"

// now we can restore our edits
patch(lib, edits);

console.log(lib[0][1][2][2]) // "edited"

// --------------------------------------------------------

// to prevent forgetting to track the updates you can use utility function
// to make changes to your 'lib' array

function update(arr, path, value) {
  applyEdit(arr, path, value);
  edits[path] = value;
}

Original answer:

One possible solution would be to have an object where you track all the changes. const edits = {}; And when you update your object with lib[x][1][y][6] = "no"; you also save your edit:

edits[`${x}.1.${y}.6`] = "no";

Basically you just create a string holding the path of the change.

After you loader your edits object from file you can apply all changes to the original array with the following code:

import { set } from 'lodash';

function patch(arr, edits) {
  Object.keys(edits).forEach(key => {
    const newValue = edits[key];
    set(arr, key, newValue);
  });
}

Answer 3

Assuming that the paths in libVars are the same in lib then you could simply record the changes into an array of functions that you reapply on your lib object later on:

const data = {foo: 'aaa', bar: 'bbb'};

console.log('data before:', data);

const updates = [];

updates.push(obj => {
  obj.foo = 'foo';
});

updates.push(obj => {
  obj.bar = 'bar';
});

updates.forEach(fn => fn(data));

console.log('data after:', data);

Answer 4

This may be handled with emulated autovivification using Proxy. See:

https://en.wikipedia.org/wiki/Autovivification

Autovivification and Javascript