1

The Setup:

Hello, my org has built an Angular PWA for our users to work offline in the field on their corporate Surface Go devices. This PWA needs to review and edit various data while users are disconnected, so we researched and settled on working with the IndexedDB (Idb) in the browser. Users have the ability to download various files, PDFs, and images while in the office, go offline and into the field, capture new photos within the app (Idb), and upload this all back to our servers upon return.

We store large JSON and blob objects in the Idb to keep track of everything and users have a good workflow to follow for downloading, fielding, and uploading. We also keep a close watch on the Storage within the browser to ensure the Surfaces do meet the limits.

The PWA is loaded in MS Edge (Version 96.0.1054.62 (Official build) (64-bit)) on Windows and the PWA is installed onto their desktop to function like a native app. The Surfaces have 8GB of RAM and plenty of hard drive space for storage.

The Problem:

This has all worked great for a few months but users have now started experiencing issues with the PWA crashing upon navigating to our app. The browser is opened, navigate to the URL, the app loads for 2-3 seconds, and then the entire browser crashes (all tabs and windows). Users have reported uploading and storing photos (3MB each) when the first crash happens. Users are then out of commission and can no longer access the app by any means, so there defiantly seems like a tipping point somewhere.

We have narrowed it down to a memory (RAM) issue with the browser. When the app starts up and establishes a connection to the Idb, the memory spikes dramatically based on the amount of data stored within it. Looking at the local File System in Windows the users that have been affected have upwards of 4.5GB+ stored in their Idb (this is everything).

Browser Memory Spike

Loading the browser .exe into Visual Studio and navigating to the app we can see the result. As soon as the call the indexeddb.open happens, a spike in memory is shown. Chrome memory spike

Chrome (in this instance) throws memory errors.

We have also tried to increase the amount of memory available to the browser by following these posts with no difference. We have been able to copy out the Idb files from the affected users (C:\Users{user}\AppData\Local\Microsoft\Edge\User Data\Default\IndexedDB), swapped them into our machines, and been able to reproduce the crashing locally.

enter image description here

System memory shows the same spike, but there is obviously a lot more memory to give.

As it stands, our users are dead in the water and unable to open the app and unable to access their data. The raw Idb files are unreadable and every solution we have taken to extract the data (https://github.com/amyboyd/indexeddb-to-json) just fails in the same way.

Questions:

  1. Can we allocate more memory to the browser through some other means?
  2. Can we extract the data from the Idb without the browser?
  3. Can we avoid the memory issues when opening the Idb some other way?
jcc
  • 1,137
  • 1
  • 12
  • 31
  • Just connecting to a database shouldn't use much memory. I've put tons of data in IndexedDB without seeing this problem. Are you absolutely sure this is the problem, not some other part of your code? If so, it could be a browser bug related to the specific type of data you're storing. If you can make a minimal reproduction of the problem, you should submit a bug report. – dumbmatter Dec 24 '21 at 17:49
  • I agree, nothing we have read indicates that this should be happening like this. We are storing blobs (pdf), json data, and base64 encoded strings. We have found issues related to the base64 strings as they are very large (3-4MB) and there can be hundreds. Some of the PDFs are over 100MB but in general those seem OK. I'm thinking this is more and more related to encoding all the base64 strings... – jcc Dec 26 '21 at 01:05
  • I'm author of GroupSpeedDial and some of my Chromium users face this exact problem. Upon extension start (or browser start since I open DB in background script) there is a HUGE memory spike - often several GB which then goes down if browser doesn't crash. In my IndexedDB I'm storing small images as Blobs and some small but complex JSON objects. But total size of my IndexedDB is only few MB, like 20MB. The Chrome "Task manager" reports "Browser" process taking all of that memory. What's strange is that the memory spike is different every time. So this is gonna be some race condition :(. Help? – icl7126 Mar 08 '22 at 09:51

1 Answers1

1

OK, so months later we were able to implement a different approach that goes around the IndexedDB and uses the File System API. This has its own drawbacks but this fixed all my issues by getting raw files out of the IndexedDB.

Related to the IndexedDB itself, we found some very specific issues around the images we were storing there, as they were all being base64 encoded and stored as strings. It seems that when the IndexedDB stores these large strings (3-4MB) it has to decode them when it wakes up to correctly render them on demand at a later time.

Based on the number of large strings it has to do this for when making the initial connection to the database, it causes a huge memory spike (decoding hundreds of base64 strings) and ultimately crashes the browser as the IndexedDB and browser engine just completely crashes.

We were not able to dig too deep into how and what the IndexedDB does under the covers as some proprietary magic from the creators at Google? I'm hoping someone with more knowledge on standards at the WWWC can chime and provide more data on this. I would love to know how it works under the covers and why we can't access the raw data on the file system.

TLDR; Don't store large base64 encoded strings in the IndexedDB.

jcc
  • 1,137
  • 1
  • 12
  • 31