Getting KeyStoreException and GeneralSecurityException by using EncryptedSharedPreferences, how can I solve those?

Question

Background

On one of the apps I work on, I store important stuff (tokens) into EncryptedSharedPreferences (taken from here and here):

/** a hardware-encrypted based shared preference (for the values).
 * Note that it is a bit slow, so it's better to always use it in a background thread.
 * Also, avoid having it being backed-up in the manifest, as it's hardware based and will become useless: https://stackoverflow.com/a/63795282/878126*/
object SecuredSharedPreferences {
    private var cachedDefaultSharedPreferences: SharedPreferences? = null

    /**warning: using this function can take some time (249 ms on Pixel 4, for example). Very recommended to avoid calling it on UI thread */
    @WorkerThread
    fun getDefaultSecuredSharedPreferences(context: Context): SharedPreferences {
        if (cachedDefaultSharedPreferences != null)
            return cachedDefaultSharedPreferences!!
        synchronized(this) {
            if (cachedDefaultSharedPreferences != null)
                return cachedDefaultSharedPreferences!!
            cachedDefaultSharedPreferences = getSecuredSharedPreferences(context, context.packageName + "_secured_preferences")
        }
        return cachedDefaultSharedPreferences!!
    }

    @WorkerThread
    private fun getSecuredSharedPreferences(context: Context, fileName: String): SharedPreferences {
        val masterKey = MasterKey.Builder(context, MasterKey.DEFAULT_MASTER_KEY_ALIAS).setKeyScheme(MasterKey.KeyScheme.AES256_GCM).build()
        return EncryptedSharedPreferences.create(context, fileName, masterKey,
                EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV,
                EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM
        )
    }
}

gradle:

implementation 'androidx.security:security-crypto:1.1.0-alpha03'

The problem

I've noticed 2 bugs being reported via Crashlytics when using this code (reported here):

First one is of the MasterKey.Builder line of GeneralSecurityException :

Fatal Exception: java.security.GeneralSecurityException: Keystore operation failed
       at androidx.security.crypto.MasterKeys.generateKey(MasterKeys.java:146)
       at androidx.security.crypto.MasterKeys.getOrCreate(MasterKeys.java:97)
       at androidx.security.crypto.MasterKey$Builder.buildOnM(MasterKey.java:357)
       at androidx.security.crypto.MasterKey$Builder.build(MasterKey.java:314)
...
Caused by java.security.ProviderException: Keystore operation failed
       at android.security.keystore.AndroidKeyStoreKeyGeneratorSpi.engineGenerateKey(AndroidKeyStoreKeyGeneratorSpi.java:372)
       at javax.crypto.KeyGenerator.generateKey(KeyGenerator.java:612)
       at androidx.security.crypto.MasterKeys.generateKey(MasterKeys.java:142)
       at androidx.security.crypto.MasterKeys.getOrCreate(MasterKeys.java:97)
       at androidx.security.crypto.MasterKey$Builder.buildOnM(MasterKey.java:357)
       at androidx.security.crypto.MasterKey$Builder.build(MasterKey.java:314)

Second one is on EncryptedSharedPreferences.create line of KeyStoreException, and occurs more often and for more users :

Fatal Exception: java.security.KeyStoreException: the master key android-keystore://_androidx_security_master_key_ exists but is unusable
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewMasterKey(AndroidKeysetManager.java:275)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:236)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:155)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)
...
Caused by java.security.UnrecoverableKeyException: Failed to obtain information about key
       at android.security.keystore.AndroidKeyStoreProvider.loadAndroidKeyStoreSecretKeyFromKeystore(AndroidKeyStoreProvider.java:282)
       at android.security.keystore.AndroidKeyStoreSpi.engineGetKey(AndroidKeyStoreSpi.java:98)
       at java.security.KeyStore.getKey(KeyStore.java:825)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.<init>(AndroidKeystoreAesGcm.java:58)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.getAead(AndroidKeystoreKmsClient.java:164)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewMasterKey(AndroidKeysetManager.java:267)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:236)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:155)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)

EDIT: seems there are even more types of exceptions:

InvalidProtocolBufferException:

Fatal Exception: com.google.crypto.tink.shaded.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
       at com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1566)
       at com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1663)
       at com.google.crypto.tink.proto.Keyset.parseFrom(Keyset.java:957)
       at com.google.crypto.tink.integration.android.SharedPrefKeysetReader.read(SharedPrefKeysetReader.java:84)
       at com.google.crypto.tink.CleartextKeysetHandle.read(CleartextKeysetHandle.java:58)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.read(AndroidKeysetManager.java:328)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewKeyset(AndroidKeysetManager.java:287)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:238)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:160)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)

NullPointerException:

Fatal Exception: java.lang.NullPointerException: Attempt to invoke interface method 'android.security.keymaster.OperationResult android.security.IKeystoreService.begin(android.os.IBinder, java.lang.String, int, boolean, android.security.keymaster.KeymasterArguments, byte[], int)' on a null object reference
       at android.security.KeyStore.begin(KeyStore.java:501)
       at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:248)
       at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:109)
       at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:2977)
       at javax.crypto.Cipher.tryCombinations(Cipher.java:2884)
       at javax.crypto.Cipher$SpiAndProviderUpdater.updateAndGetSpiAndProvider(Cipher.java:2789)
       at javax.crypto.Cipher.chooseProvider(Cipher.java:956)
       at javax.crypto.Cipher.init(Cipher.java:1199)
       at javax.crypto.Cipher.init(Cipher.java:1143)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.encryptInternal(AndroidKeystoreAesGcm.java:84)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.encrypt(AndroidKeystoreAesGcm.java:72)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.validateAead(AndroidKeystoreKmsClient.java:248)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.getAead(AndroidKeystoreKmsClient.java:165)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewMasterKey(AndroidKeysetManager.java:267)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:236)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:155)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)

What I've tried

Searching the Internet, I've found clues only for the first exception (GeneralSecurityException), that it might be that it occurs for custom ROMs, as they might not implement well the hardware keys for encryption.

And indeed, looking at the devices on Crashlytics, and looking at the Android version of each, I've found that they are ahead of what I see about the latest version that was supported for them.

For the second exception, sadly, I couldn't find any explanation and no solution either. I think it might be related to recovery of apps, but it's weird as it occurs quite often. On reddit (here), someone wrote that in case of such an exception, he chose to wrap the initialization of EncryptedSharedPreferences with "clear all data if fails" and bite the bullet. Also suggested it might be related to having android:allowBackup being disabled (and indeed it is).

No idea about the rest.

The question

Why do these exceptions occur? What can I do against them?

Is clear-data the only thing that can be done? I'm not even sure it really helps, because if I choose to have it, it means the crash report will be gone each time it's about to happen...

Is it related to android:allowBackup being disabled?

Hi, i am wondering if you still get the same error on using the stable version of `androidx.security:security-crypt`? instead of alpha, use 1.0.0 just to make sure if it still occurs on that version :) — Ric17101, Jan 05 '21 at 03:28
@Ric17101 I don't see a stable one: https://mvnrepository.com/artifact/androidx.security/security-crypto?repo=google. I think `1.0.0-rc03` is latest of 1.0.0... Even then, when using `1.0.0` or what I've found, `MasterKey` class doesn't exist... It's available only later — android developer, Jan 06 '21 at 09:54
Then, i think you could use https://developer.android.com/jetpack/androidx/releases/security#security-crypto-1.0.0-rc03 instead. I am being told to not use alpha on production since it may really affect many modules and features on the app. — Ric17101, Jan 06 '21 at 11:39
`1.1.0-alpha01` (`-02` and `-03` fails) and `android:allowBackup="false"`worked for me, at least in the device it was throwing... I believe it's kind of random. — GuilhE, Aug 13 '21 at 11:51

Anoop M Maddasseri · Answer 1 · 2022-03-23T05:42:20.453

Beware about this active glitch prior to using Jetpack Security EncryptedSharedPreferences ( even stable ), It's hitting on our crash-free sessions hard mostly from exotic devices - https://issuetracker.google.com/issues/176215143?pli=1

The only dirty workaround is - https://github.com/google/tink/issues/535#issuecomment-912661574

I'll update the answer once I find an amicable solution. Here is the workaround solution -

/**
 * A builder for creating an encrypted shared preference class.
 */
private const val KEYSTORE_PROVIDER = "AndroidKeyStore"
private const val SHARED_PREFS_FILENAME = "TVPrefs"

@KoinApiExtension
class EncryptedSharedPreferenceBuilder(var context: Context) : KoinComponent {
    private val reporter: Reporter by inject()

    private val masterKeyAlias =
        MasterKey.Builder(context).setKeyScheme(MasterKey.KeyScheme.AES256_GCM).build()

    fun build(): SharedPreferences {
        return try {
            createSharedPreferences()
        } catch (gsException: GeneralSecurityException) {
            reporter.logException(gsException)
            Timber.d("EncryptedSharedPref: Error occurred while create shared pref=$gsException")
            // There's not much point in keeping data you can't decrypt anymore,
            // delete & re-create; user has to start from scratch
            deleteSharedPreferences()
            createSharedPreferences()
        }
    }

    private fun createSharedPreferences() = EncryptedSharedPreferences.create(
        context,
        SHARED_PREFS_FILENAME,
        masterKeyAlias,
        EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV,
        EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM
    )

    // Clearing getSharedPreferences using default Preference wrapper.
    // This is to work around any key-mismatches that may happen.
    fun clearSharedPreference() {
        context.getSharedPreferences(SHARED_PREFS_FILENAME, Context.MODE_PRIVATE).edit().clear()
            .apply()
    }

    // Workaround [https://github.com/google/tink/issues/535#issuecomment-912170221]
    // Issue Tracker - https://issuetracker.google.com/issues/176215143?pli=1
    private fun deleteSharedPreferences() {
        try {
            val sharedPrefsFile =
                File("${context.filesDir.parent}/shared_prefs/$SHARED_PREFS_FILENAME.xml")

            // Clear the encrypted prefs
            clearSharedPreference()

            // Delete the encrypted prefs file
            if (sharedPrefsFile.exists()) {
                val deleted = sharedPrefsFile.delete()
                Timber.d("EncryptedSharedPref: Shared pref file deleted=$deleted; path=${sharedPrefsFile.absolutePath}")
            } else {
                Timber.d("EncryptedSharedPref: Shared pref file non-existent; path=${sharedPrefsFile.absolutePath}")
            }

            // Delete the master key
            val keyStore = KeyStore.getInstance(KEYSTORE_PROVIDER)
            keyStore.load(null)
            keyStore.deleteEntry(MasterKey.DEFAULT_MASTER_KEY_ALIAS)
        } catch (e: Exception) {
            Timber.d("EncryptedSharedPref:  Error occurred while trying to reset shared pref=$e")
        }
    }
}

This seems to work for most of the cases. Still for some rare cases it would throw an exception. — android developer, Apr 16 '22 at 21:15
Thanks ! It worked for me but I also had to catch IOException (super class of com.google.crypto.tink.shaded.protobuf.InvalidProtocolBufferException). — Showpath, Jul 27 '22 at 09:22
Thanks for the workaround, although I'd replace `File("${context.filesDir.parent}/shared_prefs/$SHARED_PREFS_FILENAME.xml").delete()` with `context.deleteSharedPreferences(SHARED_PREFS_FILE_NAME)`, much safer to delete it via the official Context API rather than hardcoding the path, you never know if some peculiar device with a custom Android changes the path, or if official Android changes it in future versions. — M.Ed, Aug 21 '23 at 09:51

score 4 · Answer 2 · answered Jan 05 '21 at 05:26

4

I think your instinct about android:allowBackup=false being the culprit is correct. I experienced a similar issue (encrypted shared preferences throwing errors on app update when allowBackup was set to false). In my case I managed to reproduce by installing, logging in (to set the encrypted preference), and then updating to a more recent version of the app and attempting to read the preference. I ended up working around the issue by removing android:allowBackup=false

answered Jan 05 '21 at 05:26

davehenry

1,026
9
24

1

What if the team manager wants it to avoid backup, though? Is it possible to include only what this feature needs, excluding the rest ? – android developer Jan 05 '21 at 21:04
Then you're out of luck. No, I don't believe so. – davehenry Jan 06 '21 at 02:34
You had this issue too (various, rare exceptions when using this code) , you say, and that after you've removed this `android:allowBackup=false` , it never happened again? – android developer Jan 06 '21 at 09:13
Not quite - I was using a different implementation of encrypted preferences that also relied on the keystore. The types of errors I saw were similar, and yeah removing allowBackup=false resolved it. – davehenry Jan 06 '21 at 16:54
1

Well if it uses the "MasterKey" and "EncryptedSharedPreferences", it should probably rely on the same things, no? – android developer Jan 06 '21 at 20:18
Yeah I'd expect so – davehenry Jan 06 '21 at 23:02
2

This is a stupid problem android/google needs to fix. – WilliaGuy Mar 16 '21 at 01:11
Problem begins when you need to use android:allowBackup=true when, for example you don't want to generate new deviceId each time after uninstalling and installing the app again. – polis Jul 23 '21 at 09:46
unfortunately, this is not an option if your app stores some sensitive user data like card number, keys etc. – stdout Nov 03 '21 at 10:19
Unfortunately, removing `allowBackUp` doesn't seem to fix this problem. There are reports in https://issuetracker.google.com/issues/176215143 about this being the case. – Christian García Mar 14 '22 at 10:14

Getting KeyStoreException and GeneralSecurityException by using EncryptedSharedPreferences, how can I solve those?

Background

The problem

What I've tried

The question

2 Answers2

Linked