Sync storage, max quota bytes per item and chunked data storage

Question

I'm using a modified version of this code (Update: that answer has since been updated to use correct code, but this question still carries value since it contains relevant test cases and discussions for this problem) to store a single object after stringification in chunked keys inside of sync storage. Note that sync storage has a maximum quota size per item. So, I have those maxLengthPerItem and maxValueLength variables.

function lengthInUtf8Bytes(str) {
    // by: https://stackoverflow.com/a/5515960/2675672
    // Matches only the 10.. bytes that are non-initial characters in a multi-byte sequence.
    var m = encodeURIComponent(str).match(/%[89ABab]/g);
    return str.length + (m ? m.length : 0);
}

function syncStore(key, objectToStore, callback) {
    var jsonstr = JSON.stringify(objectToStore), i = 0, storageObj = {},
        // (note: QUOTA_BYTES_PER_ITEM only on sync storage)
        // subtract two for the quotes added by stringification     
        // extra -5 to err on the safe side
        maxBytesPerItem = chrome.storage.sync.QUOTA_BYTES_PER_ITEM - NUMBER, 
        // since the key uses up some per-item quota, use
        // "maxValueBytes" to see how much is left for the value
        maxValueBytes, index, segment, counter; 

    console.log("jsonstr length is " + lengthInUtf8Bytes(jsonstr));

    // split jsonstr into chunks and store them in an object indexed by `key_i`
    while(jsonstr.length > 0) {
        index = key + "_" + i++;
        maxValueBytes = maxBytesPerItem - lengthInUtf8Bytes(index);

        counter = maxValueBytes;
        segment = jsonstr.substr(0, counter);           
        while(lengthInUtf8Bytes(segment) > maxValueBytes)
            segment = jsonstr.substr(0, --counter);

        storageObj[index] = segment;
        jsonstr = jsonstr.substr(counter);
    }
    // later used by retriever function
    storageObj[key] = i;
    console.log((i + 1) + " keys used (= key + key_i)");
    // say user saves till chunk 20 in case I
    // in case II, user deletes several snippets and brings down
    // total no. of "required" chunks to 15; however, the previous chunks
    // (16-20) remain in memory unless they are "clear"ed.
    chrome.storage.sync.clear(function(){                       
        console.log(storageObj);
        console.log(chrome.storage.sync);
        chrome.storage.sync.set(storageObj, callback);          
    });
}

The problem is in this line:

maxLengthPerItem = chrome.storage.sync.QUOTA_BYTES_PER_ITEM - NUMBER,

The problem is that 5 is the minimum NUMBER for which there's no error. Here's the sample code you can use to test my theory:

var len = 102000,
    string = [...new Array(len)].map(x => 1).join(""),
    Data = {
        "my_text": string
    },
    key = "key";

syncStore(key, Data, function(){
    console.log(chrome.runtime.lastError && chrome.runtime.lastError.message);
});

Using 4 yields MAX_QUOTA_BYTES_PER_ITEM exceed error. You can yourself adjust the value of len (to 20000, 60000 < 102000, etc.) to check my theory.

Question:

Why is the current method requiring exactly 5 as the minimum value? I know there's two quotes for stringification, but what about the other 3 characters? Where'd they come from?

Additionally, I've noticed that in textual Data like this one, even 5 does not work. In the specific case above, minimum NUMBER required is 6.

Clarification:

The point of my question is not what are the other means to store data in sync.

The point of my question is why is the current method requiring exactly 5 (And why that textual data requires a 6.) Imho, my question is very specific and surely does not deserve a close vote.

Update: I've added new code which stores data based on measurement of length of UTF-8 bytes, but it still does not provide desirable results. I've also added code to more easily test my theory.

Answer 1

The problem is that Chrome applies JSON.stringify to each string chunk before storing it, which adds three \ characters to the first string (which, added to the known 2 for outer quotes, makes a full 5). This behavior is noted in the Chromium source code: Calculate the setting size based on its JSON serialization size (and the implementation does indeed compute size based on key.size() + value_as_json.size()).

That is, the value in key_0 is the string

{"my_text":"11111111...

But it is stored as

"{\"my_text\":\"11111111..."

The reason you need to account for the two outer quotes is the same reason you need to account for added slashes. Both are indicative of the output of JSON.stringify operating on a string input.

You can confirm that escape-slashes are the issue by doing

var jsonstr = JSON.stringify(objectToStore).replace(/"/g,"Z")

And observing that the required NUMBER offset is 2 instead of 5, because {Zmy_textZ:Z11111... does not have extra slashes.

I haven't looked closely, but the Lorem text contains a newline and a tab (see: id faucibus diam.\), which your JSON.stringify (correctly) turns into \n\t but then Chrome's additional stringify further expands to \\n\\t, for an extra 2 bytes you do not account for. If that gets chunked with two other quotes or other escapable characters, it could cause a chunk with 4 unaccounted-for bytes.

The solution here is to account for the escaping that Chrome will do upon storage. I'd suggest applying JSON.stringify to each segment when evaluating if it's too big, so that the correct number of bytes will be consumed by the chunking algorithm. Then, once you decide on a size that will not cause problems, even after being double-stringifed, consume that many bytes from the regular string. Something like:

while(lengthInUtf8Bytes(JSON.stringify(segment)) > maxValueBytes)
    ...

Note that this will automatically account for the two bytes from outer quotes, so there's no need to even have a QUOTA_BYTES_PER_ITEM - NUMBER computation. In the terms you've presented it, with this approach, the NUMBER is 0.

Answer 2

For some reason, the technique only works when we do this:

while(lengthInUtf8Bytes(JSON.stringify(JSON.stringify(segment))) > maxValueBytes)

here's a paste containing data that you can use to compare both this and @apsiller's original approach (and verify the fact that only the above approach works).

Here's the code I used to test all this stuff

I am not accepting either answer yet since neither of them provides an acceptable logic as to why only the above approach is working.

Answer 3

After carefully reading through this thread I finally was able to understand where the extra bytes come from. apsillers actually reference the part from the chromium code that holds the answer: key.size() + value_as_json.size()

You have to account for the side of the key as well. So the working accurate check is: while((lengthInUtf8Bytes(JSON.stringify(segment)) + key.length) > maxValueBytes)