Can I make a "Virtual Array" in JavaScript?

Question

I'm calling a JavaScript function that wants an array of things to display. It displays a count, and displays the items one by one. Everything works when I pass it a normal JavaScript array.

But I have too many items to hold in memory at once. What I'd like to do, is pass it an object with the same interface as an array, and have my method(s) be called when the function tries to access the data. And in fact, if I pass the following:

var featureArray = {length: count, 0: func(0)};

then the count is displayed, and the first item is correctly displayed. But I don't want to assign all the entries, or I'll run out of memory. And the function currently crashes when the user tries to display the second item. I want to know when item 1 is accessed, and return func(1) for item 1, and func(2) for item 2, etc. (i.e., delaying the creation of the item until it is requested).

Is this possible in JavaScript?

Answer 1

Yes, generating items on the go is possible. You will want to have a look at Lazy.js, a library for producing lazily computed/loaded sequences.

However, you will need to change your function that accepts this sequence, it will need to be consumed differently than a plain array.

---

If you really need to fake an array interface, you'd use Proxies. Unfortunately, it is only a harmony draft and currently only supported in Firefox' Javascript 1.8.5.

Assuming that the array is only accessed in an iteration, i.e. starting with index 0, you might be able to do some crazy things with getters:

var featureArray = (function(func) {
    var arr = {length: 0};
    function makeGetter(i) {
        arr.length = i+1;
        Object.defineProperty(arr, i, {
            get: function() {
                var val = func(i);
                Object.defineProperty(arr, i, {value:val});
                makeGetter(i+1);
                return val;
            },
            configurable: true,
            enumerable: true
        });
    }
    makeGetter(0);
    return arr;
}(func));

However, I'd recommend to avoid that and rather switch the library that is expecting the array. This solution is very errorprone if anything else is done with the "array" but accessing its indices in order.

Answer 2

If I understand correctly, this would help:

var object = {length: count, data: function (whatever) {
    // create your item
}};

Then, instead of doing array[1], array[2], et cetera, you'd do object.data(1), object.data(2), and so on.

Answer 3

Since there seems to be a constraint that the data must be accessed using array indexing via normal array indexing arr[index] and that can't be changed, then the answer is that NO, you can't override array indexing in Javascript to change how it works and make some sort of virtual array that only fetches data upon demand. It was proposed for ECMAScript 4 and rejected as a feature.

See these two other posts for other discussion/confirmation:

How would you overload the [] operator in Javascript

In javascript, can I override the brackets to access characters in a string?

The usual way to solve this problem would be to switch to using a method such as .get(n) to request the data and then the implementor of .get() can virtualize however much they want.

---

P.S. Others indicate that you could use a Proxy object for this in Firefox (not supported in other browsers as far as I know), but I'm not personally familiar with Proxy objects as it's use seems rather limited to code that only targets Firefox right now.

Answer 4

Thank you to everyone who has commented and answered my original question - it seems that this is not (currently) supported by JavaScript.

I was able to get around this limitation, and still do what I wanted. It uses an aspect of the program that I did not mention in my original question (I was trying to simplify the question), so it is understandable that other's couldn't recommend this. That is, it doesn't technically answer my original question, but I'm sharing it in case others find it useful.

It turns out that one member of the object in each array element is a callback function. That is (using the terminology from my original question), func(n) is returning an object, which contains a function in one member, which is called by the method being passed the data. Since this callback function knows the index it is associated with (at least, when being created by func(n)), it can add the next item in the array (or at least ensure that it is already there) when it is called. A more complicated solution might go a few ahead, and/or behind, and/or could cleanup items not near the current index to free memory. This all assumes that the items will be accessed consecutively (which is the case in my program).

E.g.,

1) Create a variable that will stay in scope (e.g., a global variable).

2) Call the function with an object like I gave as an example in my original question:

var featureArray = {length: count, 0: func(0)};

3) func() can be something like:

function func(r) {
    return {
        f : function() {featureArray[r + 1] = func(r + 1); DoOtherStuff(r); }
    }
}

Assuming that f() is the member with the function that will be called by the external function.