I'm creating a Chrome extension that stores a lot of data in chrome.storage.local. By "a lot," I mean that when I run something like this:
chrome.storage.local.getBytesInUse(i => console.log(i))
...I print 150873495, indicating that I'm storing about 150 MB. I would like to be able to store more than this. I'm aware that this is a lot of data for a Chrome extension, but it will be running on my own machine.
I previously posted How can I retrieve lots of data in chrome.storage.local?, and that issue was solved by a bug fix within Chrome. Now I have another problem.
I would like to transfer all of the data in chrome.storage.local to some sort of text file (such as JSON). However, when I run something as simple as this:
chrome.storage.local.get(
null,
result => {
console.log("Records retrieved");
}
);
...the memory usage of my extension (found in the Chrome task manager) spikes to about 2.4 GB, more than an order of magnitude greater than the 150 MB that the data takes up on the disk in chrome.storage.local. Let's say I then want to save this data to a file, using the method described in Chrome Extension: Local Storage, how to export. I need to call JSON.stringify:
chrome.storage.local.get(
null,
result => {
console.log("Records retrieved");
const json = JSON.stringify(result);
console.log("JSON string length: " + json.length);
}
);
It never gets to the "JSON string length" comment because the JSON.stringify call causes memory usage to go beyond the 4 GB memory limit, and the extension crashes. I have a couple of questions:
- Is there a way to save this 150 MB of storage without using 4 GB of memory?
- Is there an easy way to save this data in chunks? Given the asynchronous nature of JavaScript and
chrome.storage, it seems like building a safe way to do this would be tricky.
Edit
As requested by @wOxxOm, here's some code used to generate fake data:
function randomString() {
return Math.floor(Math.random() * 0xFFFFFFFF).toString(16);
}
for (let i = 0; i < 70000; i++) {
const item = {time: Date.now()};
for (let j = 0; j < 100; j++ ) {
item[randomString()] = randomString();
}
const recordName = "record-" + Date.now() + "-" + randomString();
const items = {};
items[recordName] = item;
chrome.storage.local.set(items);
}
This seems to take up about 160 MB in chrome.storage.local, but about 2.7 GB in memory.
