Adding many items to indexedDB

Question

I am looking for best practices when it comes to adding many items to an indexedDB. The upper bound of the number of items is usually around 100k, and each item is a js object usually around 350 bytes.

I maintain a list of pending items (actions) to be taken on the database (each action is either "add", "put", or "delete"), and each item could belong to a different object store.

I simply take the first item, apply it to the appropriate objectStore, and once that's done (either successfully or not), I move on to the next item.

Is there a better (more efficient) way to do this? And are there any scoping issues w.r.t. transaction lifetime I need to be concerned with in the snippet below?

function flushPendingDbActions() {
    var transaction = db.transaction(storeNames, "readwrite");

    var addNext = function() {
        var nextItem = pendingDbActions[0];
        if (nextItem) {
            pendingDbActions.shift();

            var objectStore = transaction.objectStore(nextItem.store),
                params,
                request;

            switch(nextItem.action) {
            case 'add':
            case 'put':
                params = nextItem;
                break;
            case 'delete':
                params = nextItem.key;
                break;
            }

            request = objectStore[nextItem.action](params);
            request.onsuccess = request.onerror = addNext;
        }
    };

    addNext();
}

Answer 1

Looks alright to me, but a couple things:

• I'd stop forward iteration on error. An error event sometimes indicates something is really wrong, and that all later requests on that transaction will probably also fail, and by continuing on the error path you are just going to cause N more errors, where N is the remaining number.
• I'd use a stack or lifo access pattern. Append items onto an array, then pop them off to advance. Avoid using shift and fifo-queue like access, it is just bad perf in js because shift does a ton more work than pop. This is nitpicking of course but you kind of asked about efficiency. Kinda only matters if dealing with a large N.
• Depending on the number of items, you can run into a stack overflow kind of error, because each request is a pending function on the so-called stack, which from your code looks like it accumulates unbounded. In this case you may want to batch requests into chunks of like 100 or 1000 or something to avoid that issue ever arising, but this only matters if you are even getting above the threshold number, and the particular platform where the code is running (e.g. on mobile might run into trouble).
• Then there is a more subtle async issue. Right now you are waiting on each request to complete before issuing the next request. Why? If the requests are logically independent, fire them all off immediately, and wait on the transaction to complete (which completes whenever the last pending request completes). You generally only need to use the pattern you're using now, where each action waits for all prior actions to settle, if each subsequent request's logic changes based on the prior requests. So you're introducing idleness where none is needed, and basically speed-limiting yourself for no reason, again, provided that the requests are independent.

Edit - added content:

According to https://www.w3.org/TR/IndexedDB-2/#transaction-lifetime-concept:

Unless otherwise defined, requests must be executed in the order in which they were made against the transaction. Likewise, their results must be returned in the order the requests were placed against a specific transaction.

Therefore, you should be able to fire off your requests concurrently instead of serially, which should avoid the TransactionInactiveError. This is true even if you fire off requests that do the equivalent of add item > delete item.

Then you run into the stack overflow issue with too many pending requests because requests are located in the stack, and then you should consider buffering in that case. So take the pendingactions array and process it in chunks. Take let's say 1000 at a time, pass 1000 to a processchunk helper function that does fires off 1000 concurrent requests, wait for it to complete, and then process the next chunk, up until all chunks processed.