Using Android Room, what is the maximum size of a blob and where is this documented?

Question

I'm using Android Room and storing ByteArray data as BLOBs. Above a certain size, an attempt to retrieve a row will fail with the following message:

E/CursorWindow: The size of (0, 2) is too big (4194304), so replace the value to NULL

and the actual cause of the crash is:

    java.lang.NullPointerException: Parameter specified as non-null is null: method kotlin.jvm.internal.Intrinsics.checkNotNullParameter, parameter bytes

So, above some size the CursorWindow cannot load the blob, returns null, and the DAO method throws NPE because the entity field is not nullable.

I can appreciate that there would be some upper bound on blob size in Sqlite/Room. Experimentally I have determined that it is 4194304 (4096*1024) for my particular setup (at least under certain circumstances).

My questions are: where is this documented? Could it vary by device, Room/Sqlite version, etc/. Can I determine this value at runtime?

I would like to enforce appropriate byte array limits prior to data insertion so this error never occurs, and I am looking for a principled/reliable way to determine the actual limit.

UPDATE

I think @MikeT's answer is the best we can do. The maximum row size we can retrieve using android.database.sqlite is essentially undocumented; most put it at ~2MB but for some reason I (some of the time) peg it at ~4MB using Android 12. Here is a version of @MikeT's answer in Kotlin that should work whether your setup throws an exception on oversized rows or (like mine) returns a null value for the column (which, when using a Room entity with a non-nullable field, will result in a NPE thrown by the dao method).

    fun checkCursorWindowSize() {
        val context = this
        val TEST_TABLE_NAME = "tcw"
        val TEST_DATABASE_NAME = "tcw"
        val TEST_COLUMN_NAME = "testblob"
        val reducer = 1024 * 1024 / 16 /* size to reduce BLOB by per iteration */
        val allowance =
            1 /* size to reduce BLOB by for all extractions (reason for use probably best to not be on a power of 2 boundary????) */
        val ba = ByteArray(1024 * 1024 * 4) /* initial size of BLOB  when storing*/
        var csr: Cursor
        val tcwDbPath: String = context.getDatabasePath(TEST_DATABASE_NAME).getPath()
        val dbFile: File = context.getDatabasePath(TEST_DATABASE_NAME)
        val dbDirectory: File = dbFile.getParentFile()
        /* If the database exists then delete it = should never exist */if (dbFile.exists()) {
            dbFile.delete()
        }
        /* Ensure that the databases directory exists - otherwise openOrCreate fails */if (!dbDirectory.exists()) {
            Log.d(
                "DEBUGINFO",
                "Database directory " + dbDirectory.getPath()
                    .toString() + " does not exist - Creating"
            )
            dbDirectory.mkdirs()
        }

        /* Finally create the database */
        val db = SQLiteDatabase.openOrCreateDatabase(tcwDbPath, null, null)
        /* Do everything inside a transaction to reduce writes */db.beginTransaction()
        /* Create the table into which the BLOB will be loaded */db.execSQL("CREATE TABLE IF NOT EXISTS $TEST_TABLE_NAME ($TEST_COLUMN_NAME BLOB)")
        /* Insert the over-sized BLOB */
        val cv = ContentValues()
        cv.put(TEST_COLUMN_NAME, ba)
        db.insert(TEST_TABLE_NAME, null, cv)
        /* Prepare to repeatedly extract an eve decreasing part of the blob */
        var retrieveSize = ba.size /* first try to retrieve entire BLOB */
        var caught = 0 /* the number of caught exceptions */
        /* Try 1024 attempts at getting smaller BLOB */
        for (i in 0..1024) {
            Log.d("DEBUG", "Iteration is " + (i + 1) + ". Retrieve Size is " + retrieveSize)
            try {
                /* Build the query to extract part of the blob (using the SQLite substr function) according to
                    the current retrieve size less the allowance
                    only getting 1 row (should only be 1 row)
                    NOTE!! retrieving an oversize BLOB does not fail here, rather the failure is when attempting to read the data (getBlob method)
                 */
                csr = db.query(
                    TEST_TABLE_NAME,
                    arrayOf("substr(" + TEST_COLUMN_NAME + ",1," + (retrieveSize - allowance) + ") AS " + TEST_COLUMN_NAME),
                    null,
                    null,
                    null,
                    null,
                    null,
                    "1"
                )
                var test: ByteArray?=null
                while (csr.moveToNext()) {
                    test = csr.getBlob(0) /* get the Blob from the  1st (only) column */
                    Log.d("DEBUG", "Received value: ${test?.size}")
                }
                if (test != null) {
                    break
                }

            } catch (e: SQLException) {
                caught++
                //e.printStackTrace();
            } finally {
                retrieveSize =
                    retrieveSize - reducer /* reduce the retrieveSize ready for the next iteration (if one )*/
                if (retrieveSize < 0) break
            }
        }
        Log.d("DEBUGINFO", "Caught $caught SQlite Exceptions")
        /* Done with the database so clean up */db.endTransaction()
        db.close()
        File(tcwDbPath).delete()
        /* adjust the retrieve size as it has been adjusted in preparation for the next iteration */
        Log.d("DEBUGINFO","MaxCursorSize = " + (retrieveSize + reducer).toLong())
    }

Does this answer your question? [What is the Maximum size of the image to be stored as BLOB in sqlite database?](https://stackoverflow.com/questions/11737955/what-is-the-maximum-size-of-the-image-to-be-stored-as-blob-in-sqlite-database) — forpas, Aug 28 '22 at 17:55
Thank you - but no, because while that suggests sqlite can support blobs over 1GB in size, in the context of using Android/Room I'm hitting a much lower limit. — jwillc, Aug 28 '22 at 18:04
I have, thanks. This is appears to be a limit outside of sqlite, related to the android.database implementation. I think it should be related to the value of com.android.internal.R.integer.config_cursorWindowSize * 1024, but I'm not sure, because I've been able to retrieve values up to around 4MB. — jwillc, Aug 28 '22 at 18:38
This explanation seems helpful: https://stackoverflow.com/questions/31465069/does-android-sqlite-cursor-load-all-records-into-memory-at-once (hqt's answer). But it's not quite adding up because in certain tests I fail at around 4MB, but as near as I can tell, my value of config_cursorWindowSize is 2048kb (https://android.googlesource.com/platform/frameworks/base/+/refs/heads/android12--mainline-release/core/res/res/values/config.xml). — jwillc, Aug 28 '22 at 18:41

MikeT · Accepted Answer · 2022-08-29T10:10:42.553

Your issue is not the maximum size of a BLOB, rather it is the maximum size of a Cursor Window, which is specific to the Android SQLite API/implementation and the limit varies from 1Mb to 4Mb (if I recall correctly) depending upon the version of Android (later Versions having the larger size).

The Cursor Window is wrapped within the Cursor implementation which for Room is wrapped inside the convenience methods, so determining the size, although probably possible would likely be a difficult/complex task.

It is possible to get around the issue, which is that the Cursor Window MUST be able to store a full row (so other columns impact upon how big the Blob itself can be). You can extract parts (chunks) of the Blob and then combine them. Using chunks that are less than 1Mb (probably much less) would cover all limitations.

However, for such large amounts of data, it is recommended to store the data as a file and then store enough of the file path to be able to retrieve the file.

Here's two answers (not using Room though) that retrieve data from a Blob that is greater than the Cursor Window limit.

Do you know how I can determine the limit size for a particular version of Android (or at runtime for the version my app is on)? I found config_cursorWindowSize = 2048 in AOSP code for Android 12 (which is what I'm testing on at the moment), but that's inconsistent with the fact that I've had queries that worked up to about ~4MB per blob (or really, per row, to your point).

The following, from limited testing, (API 28 and 31) is one way BUT I suspect that this may alter (both show 2MB i.e. 2097152) depending upon other uses as CursorWindow can be constructed using a constructor where the windowSize is passed.

Here's the basic/limited java method which

creates a database and a table with just the 1 column into which an oversized BLOB (4Mb) is inserted as the only row,
then progressively tries to extract a smaller part (all at first) of the BLOB until the data can be extracted without the exception
finally it clears up the database

:-

long checkCursorWindowSize(Context context) {
        final String TEST_TABLE_NAME = "tcw";
        final String TEST_DATABASE_NAME = "tcw";
        final String TEST_COLUMN_NAME = "testblob";
        final int reducer = 1024 * 1024 / 16; /* size to reduce BLOB by per iteration */
        final int allowance = 1; /* size to reduce BLOB by for all extractions (reason for use probably best to not be on a power of 2 boundary????) */
        byte[] ba = new byte[1024 * 1024 * 4]; /* initial size of BLOB  when storing*/

        Cursor csr;
        String tcwDbPath = context.getDatabasePath(TEST_DATABASE_NAME).getPath();
        File dbFile = context.getDatabasePath(TEST_DATABASE_NAME);
        File dbDirectory = dbFile.getParentFile();
        /* If the database exists then delete it = should never exist */
        if (dbFile.exists()) {
            dbFile.delete();
        }
        /* Ensure that the databases directory exists - otherwise openOrCreate fails */
        if (!dbDirectory.exists()) {
            Log.d("DEBUGINFO", "Database directory " + dbDirectory.getPath() + " does not exist - Creating");
            dbDirectory.mkdirs();
        }

        /* Finally create the database */
        SQLiteDatabase db = SQLiteDatabase.openOrCreateDatabase(tcwDbPath,null,null);
        /* Do everything inside a transaction to reduce writes */
        db.beginTransaction();
        /* Create the table into which the BLOB will be loaded */
        db.execSQL("CREATE TABLE IF NOT EXISTS " + TEST_TABLE_NAME + " (" + TEST_COLUMN_NAME + " BLOB)");
        /* Insert the over-sized BLOB */
        ContentValues cv = new ContentValues();
        cv.put(TEST_COLUMN_NAME,ba);
        db.insert(TEST_TABLE_NAME,null,cv);
         /* Prepare to repeatedly extract an eve decreasing part of the blob */
        int retrieveSize = ba.length; /* first try to retrieve entire BLOB */
        int caught = 0; /* the number of caught exceptions */
        /* Try 1024 attempts at getting smaller BLOB */
        for (int i=0; i <= 1024; i++  ) {
            Log.d("DEBUG","Iteration is " + (i + 1) + ". Retrieve Size is " + retrieveSize);
            try {
                /* Build the query to extract part of the blob (using the SQLite substr function) according to
                    the current retrieve size less the allowance
                    only getting 1 row (should only be 1 row)
                    NOTE!! retrieving an oversize BLOB does not fail here, rather the failure is when attempting to read the data (getBlob method)
                 */
                csr = db.query(TEST_TABLE_NAME, new String[]{"substr(" + TEST_COLUMN_NAME + ",1," + (retrieveSize - allowance) + ") AS " + TEST_COLUMN_NAME},null,null,null,null,null,"1");
                while (csr.moveToNext()) {
                    byte[] test = csr.getBlob(0); /* get the Blob from the  1st (only) column */
                }
                break; /* if no Exception  then all done */
            } catch (SQLException e) {
                retrieveSize = retrieveSize - reducer; /* reduce the retrieveSize ready for the next iteration (if one )*/
                caught++;
                //e.printStackTrace();
            }
        }
        Log.d("DEBUGINFO","Caught " + caught + " SQlite Exceptions");
        /* Done with the database so clean up */
        db.endTransaction();
        db.close();
        new File(tcwDbPath).delete();
        /* adjust the retrieve size as it has been adjusted in preparation for the next iteration */
            return retrieveSize + reducer;
    }

The limited testing used the following in MainActivity :-

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    Log.d("DEBUGINFO","MaxCursorSize = " + checkCursorWindowSize(this));

}

On API 28 :- D/DEBUGINFO: MaxCursorSize = 2097152 With the whole log being:-

2022-08-29 10:58:06.798 D/DEBUG: Iteration is 16. Retrieve Size is 3211264
2022-08-29 10:58:06.802 W/CursorWindow: Window is full: requested allocation 3211263 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.802 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,3211263) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.802 D/DEBUG: Iteration is 17. Retrieve Size is 3145728
2022-08-29 10:58:06.806 W/CursorWindow: Window is full: requested allocation 3145727 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.807 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,3145727) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.807 D/DEBUG: Iteration is 18. Retrieve Size is 3080192
2022-08-29 10:58:06.810 W/CursorWindow: Window is full: requested allocation 3080191 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.810 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,3080191) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.810 D/DEBUG: Iteration is 19. Retrieve Size is 3014656
2022-08-29 10:58:06.813 W/CursorWindow: Window is full: requested allocation 3014655 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.813 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,3014655) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.813 D/DEBUG: Iteration is 20. Retrieve Size is 2949120
2022-08-29 10:58:06.818 W/CursorWindow: Window is full: requested allocation 2949119 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.819 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2949119) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.819 D/DEBUG: Iteration is 21. Retrieve Size is 2883584
2022-08-29 10:58:06.824 W/CursorWindow: Window is full: requested allocation 2883583 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.824 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2883583) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.825 D/DEBUG: Iteration is 22. Retrieve Size is 2818048
2022-08-29 10:58:06.829 W/CursorWindow: Window is full: requested allocation 2818047 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.829 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2818047) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.829 D/DEBUG: Iteration is 23. Retrieve Size is 2752512
2022-08-29 10:58:06.832 W/CursorWindow: Window is full: requested allocation 2752511 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.832 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2752511) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.832 D/DEBUG: Iteration is 24. Retrieve Size is 2686976
2022-08-29 10:58:06.837 W/CursorWindow: Window is full: requested allocation 2686975 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.838 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2686975) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.838 D/DEBUG: Iteration is 25. Retrieve Size is 2621440
2022-08-29 10:58:06.842 W/CursorWindow: Window is full: requested allocation 2621439 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.842 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2621439) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.842 D/DEBUG: Iteration is 26. Retrieve Size is 2555904
2022-08-29 10:58:06.846 W/CursorWindow: Window is full: requested allocation 2555903 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.846 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2555903) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.846 D/DEBUG: Iteration is 27. Retrieve Size is 2490368
2022-08-29 10:58:06.849 W/CursorWindow: Window is full: requested allocation 2490367 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.849 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2490367) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.849 D/DEBUG: Iteration is 28. Retrieve Size is 2424832
2022-08-29 10:58:06.853 W/CursorWindow: Window is full: requested allocation 2424831 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.853 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2424831) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.854 D/DEBUG: Iteration is 29. Retrieve Size is 2359296
2022-08-29 10:58:06.857 W/CursorWindow: Window is full: requested allocation 2359295 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.858 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2359295) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.858 D/DEBUG: Iteration is 30. Retrieve Size is 2293760
2022-08-29 10:58:06.862 W/CursorWindow: Window is full: requested allocation 2293759 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.862 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2293759) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.862 D/DEBUG: Iteration is 31. Retrieve Size is 2228224
2022-08-29 10:58:06.865 W/CursorWindow: Window is full: requested allocation 2228223 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.865 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2228223) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.865 D/DEBUG: Iteration is 32. Retrieve Size is 2162688
2022-08-29 10:58:06.870 W/CursorWindow: Window is full: requested allocation 2162687 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.870 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2162687) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.870 D/DEBUG: Iteration is 33. Retrieve Size is 2097152
2022-08-29 10:58:06.874 W/CursorWindow: Window is full: requested allocation 2097151 bytes, free space 2096720 bytes, window size 2097152 bytes
2022-08-29 10:58:06.875 E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT substr(testblob,1,2097151) AS testblob FROM tcw LIMIT 1
2022-08-29 10:58:06.875 D/DEBUG: Iteration is 34. Retrieve Size is 2031616
2022-08-29 10:58:06.881 D/DEBUGINFO: Caught 33 SQlite Exceptions
2022-08-29 10:58:06.904 D/DEBUGINFO: MaxCursorSize = 2097152

On API 31 the result is exactly the same bar the time/timings i.e. MaxCursorSize is calculated returned as 2097512

However, add another column for example and then other data has to be loaded into the CursorWindow and the result is then not a true reflection of how big the Blob can be.

hence why it could well just lead to issues if any reliance were made on the limit of what a CursorWindow can hold in total.

Thanks, and this pretty much seems like the answer, except I'm still curious how to determine the exact limit for a given configuration. Do you know how I can determine the limit size for a particular version of Android (or at runtime for the version my app is on)? I found config_cursorWindowSize = 2048 in AOSP code for Android 12 (which is what I'm testing on at the moment), but that's inconsistent with the fact that I've had queries that worked up to about ~4MB per blob (or really, per row, to your point)... — jwillc, Aug 28 '22 at 20:11
Wow this is great. I'll go ahead and mark you as the answer, but also I have some interesting discrepancies on my setup. When I run this (compile/targetSdk = 31), I don't receive any SQLExceptions; instead getBlob(0) returns null when the value is too large. And the limit is ~4MB. With a few modifications (break when `test != null`) my output in relevant part is `E/CursorWindow: The size of (0, 0) is too big (4194303), so replace the value to NULL` and `MaxCursorSize = 4128768`. I wonder why? — jwillc, Aug 29 '22 at 15:06
@jwillc Ooops I'd changed used `byte[] ba = new byte[1024 * 1024 * 5];` To 4Mb. So 4Mb with an allowance of 1 is 1 byte less 4Mb could fit. The method is by no means perfect and could certainly do with some tweaking; my meaning of its **basic/limited**. (perhaps after the no exception it should then start increasing until an exception). Why my test with 31 gave 2Mb and your use of 31 gave 4Mb is why I said ***BUT I suspect that this may alter ....*** — MikeT, Aug 29 '22 at 18:42

Using Android Room, what is the maximum size of a blob and where is this documented?

1 Answers1