How can I see hidden app data in Google Drive?

Question

I have an Android app that stores my notes in hidden app data. I want to export my notes so the question is simple:

How can I access the hidden app data in Google Drive for a specific app?

Answer 1

Indeed, Google does not let you access this hidden app-data folder directly.

But, if you can get your hands on the app's client ID/client secret/digital signature that is used for authentication against Google's servers - then yes, you can basically emulate the app and access the hidden data in your Google Drive using the Drive API.

How it works in Android

Usually, when an android application wants to access a Google API (such as Drive, Games or Google Sign-In - not all are supported) it communicates with the Google Play services client library, which in turn obtains an access token from Google on behalf of the app. This access token is then sent with each request to the API, so that Google knows who is using it and what he is allowed to do with your account (OAuth 2.0). In order to get this access token for the first time, the Google Play service sends an HTTPS POST request to android.clients.google.com/auth with these fields (along with other details):

• Token - a "master token" which identifies the Google account and basically allows full access to it
• app - the application package name, such as com.whatsapp
• client_sig - the application's digital signature (sent as SHA1)
• device - the device's Android ID
• service - the scopes (permissions) that the app wants to have

So before we can start using the Drive API in the name of a specific app, we need to know its signature and our account's master token. Fortunately, the signature can be easily extracted from the .apk file:

shell> unzip whatsapp.apk META-INF/*
       Archive:  whatsapp.apk
          inflating: META-INF/MANIFEST.MF    
          inflating: META-INF/WHATSAPP.SF    
          inflating: META-INF/WHATSAPP.DSA
shell> cd META-INF
shell> keytool -printcert -file WHATSAPP.DSA   # can be CERT.RSA or similar
       .....
       Certificate fingerprints:
       SHA1: 38:A0:F7:D5:05:FE:18:FE:C6:4F:BF:34:3E:CA:AA:F3:10:DB:D7:99
       Signature algorithm name: SHA1withDSA
       Version: 3

The next thing we need is the master token. This special token is normally received and stored on the device when a new google account is added (for example, when first setting up the phone), by making a similar request to the same URL. The difference is that now the app that's asking for permissions is the Play services app itself (com.google.android.gms), and Google is also given additional Email and Passwd parameters to log in with. If the request is successful, we will get back our master token, which could then be added to the user's app request.

You can read this blogpost for more detailed information about the authentication process.

Putting it all together

Now, we can write a code for authentication using these two HTTP requests directly - a code that can browse any app's files with any Google account. Just choose your favorite programming language and client library. I found it easier with PHP:

require __DIR__ . '/vendor/autoload.php'; // Google Drive API

// HTTPS Authentication
$masterToken = getMasterTokenForAccount("your_username@gmail.com", "your_password");
$appSignature = '38a0f7d505fe18fec64fbf343ecaaaf310dbd799';
$appID = 'com.whatsapp';
$accessToken = getGoogleDriveAccessToken($masterToken, $appID, $appSignature);

if ($accessToken === false) return;

// Initializing the Google Drive Client
$client = new Google_Client();
$client->setAccessToken($accessToken);
$client->addScope(Google_Service_Drive::DRIVE_APPDATA);
$client->addScope(Google_Service_Drive::DRIVE_FILE);
$client->setClientId("");    // client id and client secret can be left blank
$client->setClientSecret(""); // because we're faking an android client
$service = new Google_Service_Drive($client);

// Print the names and IDs for up to 10 files.
$optParams = array(
    'spaces' => 'appDataFolder',
    'fields' => 'nextPageToken, files(id, name)',
    'pageSize' => 10
);
$results = $service->files->listFiles($optParams);

if (count($results->getFiles()) == 0) 
{
    print "No files found.\n";
} 
else 
{
    print "Files:\n";
    foreach ($results->getFiles() as $file) 
    {
        print $file->getName() . " (" . $file->getId() . ")\n";
    }
}

/*
$fileId = '1kTFG5TmgIGTPJuVynWfhkXxLPgz32QnPJCe5jxL8dTn0';
$content = $service->files->get($fileId, array('alt' => 'media' ));
echo var_dump($content);
*/

function getGoogleDriveAccessToken($masterToken, $appIdentifier, $appSignature)
{
    if ($masterToken === false) return false;

    $url = 'https://android.clients.google.com/auth';
    $deviceID = '0000000000000000';
    $requestedService = 'oauth2:https://www.googleapis.com/auth/drive.appdata https://www.googleapis.com/auth/drive.file';
    $data = array('Token' => $masterToken, 'app' => $appIdentifier, 'client_sig' => $appSignature, 'device' => $deviceID, 'google_play_services_version' => '8703000', 'service' => $requestedService, 'has_permission' => '1');

    $options = array(
        'http' => array(
            'header' => "Content-type: application/x-www-form-urlencoded\r\nConnection: close",
            'method' => 'POST',
            'content' => http_build_query($data),
            'ignore_errors' => TRUE,
            'protocol_version'=>'1.1',
             //'proxy' => 'tcp://127.0.0.1:8080', // optional proxy for debugging
             //'request_fulluri' => true
        )
    );
    $context = stream_context_create($options);
    $result = file_get_contents($url, false, $context);
    if (strpos($http_response_header[0], '200 OK') === false) 
    { 
        /* Handle error */
        print 'An error occured while requesting an access token: ' . $result . "\r\n";
        return false;
    }

    $startsAt = strpos($result, "Auth=") + strlen("Auth=");
    $endsAt = strpos($result, "\n", $startsAt);
    $accessToken = substr($result, $startsAt, $endsAt - $startsAt);

    return "{\"access_token\":\"" . $accessToken . "\", \"refresh_token\":\"TOKEN\", \"token_type\":\"Bearer\", \"expires_in\":360000, \"id_token\":\"TOKEN\", \"created\":" . time() . "}";
}

function getMasterTokenForAccount($email, $password) 
{
    $url = 'https://android.clients.google.com/auth';
    $deviceID = '0000000000000000';
    $data = array('Email' => $email, 'Passwd' => $password, 'app' => 'com.google.android.gms', 'client_sig' => '38918a453d07199354f8b19af05ec6562ced5788', 'parentAndroidId' => $deviceID);

    $options = array(
        'http' => array(
            'header' => "Content-type: application/x-www-form-urlencoded\r\nConnection: close",
            'method' => 'POST',
            'content' => http_build_query($data),
            'ignore_errors' => TRUE,
            'protocol_version'=>'1.1',
             //'proxy' => 'tcp://127.0.0.1:8080', // optional proxy for debugging
             //'request_fulluri' => true
        )
    );
    $context = stream_context_create($options);
    $result = file_get_contents($url, false, $context);
    if (strpos($http_response_header[0], '200 OK') === false) 
    { 
        /* Handle error */
        print 'An error occured while trying to log in: ' . $result . "\r\n";
        return false;
    }

    $startsAt = strpos($result, "Token=") + strlen("Token=");
    $endsAt = strpos($result, "\n", $startsAt);
    $token = substr($result, $startsAt, $endsAt - $startsAt);

    return $token;
}

And finally, the results -

Files:
gdrive_file_map (1d9QxgC3p4PTXRm_fkAY0OOuTGAckykmDfFls5bAyE1rp)
Databases/msgstore.db.crypt9    (1kTFG5TmgIGTPJuVynWfhkXxLPgz32QnPJCe5jxL8dTn0)
16467702039-invisible (1yHFaxfmuB5xRQHLyRfKlUCVZDkgT1zkcbNWoOuyv1WAR)
Done.

NOTE: This is an unofficial, hacky solution, and so it might have a few problems. For example, the access token is alive only for one hour, after which it won't be refreshed automatically.

Answer 2

A working example as of September 2020

Note: this is actually an addition for Tomer's answer

Things changed since Tomer's original answer was posted. Currently, to get the master token and avoid the Error=BadAuthentication, you need two things:

• Replace Passwd field with EncryptedPasswd and encrypt its value by RSA with google public key (the exact technique was reversed by some guy) - this can be done using phpseclib.
• Make HTTPS connection to Google server with the same SSL/TLS options as in one of the supported Android systems. This includes TLS versions and exact list of supported ciphers in right order. If you change the order or add/remove ciphers you'll get Error=BadAuthentication. It took me a whole day to figure this out... Luckily, PHP >=7.2 comes with openssl-1.1.1 that has all the necessary ciphers to emulate Android 10 client.

So here is rewriten getMasterTokenForAccount() function that sets the ciphers and uses EncryptedPasswd instead of plain Passwd. And below is encryptPasswordWithGoogleKey() implementation that does the encryption.

phpseclib is necessary and can be installed with composer: composer require phpseclib/phpseclib:~2.0

function getMasterTokenForAccount($email, $password) 
{
    $url = 'https://android.clients.google.com/auth';
    $deviceID = '0000000000000000';
    $data = array('Email' => $email, 'EncryptedPasswd' => encryptPasswordWithGoogleKey($email, $password), 'app' => 'com.google.android.gms', 'client_sig' => '38918a453d07199354f8b19af05ec6562ced5788', 'parentAndroidId' => $deviceID);

    $options = array(
        'ssl' => array(
            'ciphers' => 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20:ECDH+AESGCM:DH+AESGCM:ECDH+AES:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5:!DSS'),
        'http' => array(
            'header' => "Content-type: application/x-www-form-urlencoded\r\nConnection: close",
            'method' => 'POST',
            'content' => http_build_query($data),
            'ignore_errors' => TRUE,
            'protocol_version'=>'1.1',
             //'proxy' => 'tcp://127.0.0.1:8080', // optional proxy for debugging
             //'request_fulluri' => true
        )
    );
    $context = stream_context_create($options);
    $result = file_get_contents($url, false, $context);
    if (strpos($http_response_header[0], '200 OK') === false) 
    { 
        /* Handle error */
        print 'An error occured while trying to log in: ' . $result . "\r\n";
        return false;
    }

    $startsAt = strpos($result, "Token=") + strlen("Token=");
    $endsAt = strpos($result, "\n", $startsAt);
    $token = substr($result, $startsAt, $endsAt - $startsAt);

    return $token;
}

function encryptPasswordWithGoogleKey($email, $password)
{
    define('GOOGLE_KEY_B64', 'AAAAgMom/1a/v0lblO2Ubrt60J2gcuXSljGFQXgcyZWveWLEwo6prwgi3iJIZdodyhKZQrNWp5nKJ3srRXcUW+F1BD3baEVGcmEgqaLZUNBjm057pKRI16kB0YppeGx5qIQ5QjKzsR8ETQbKLNWgRY0QRNVz34kMJR3P/LgHax/6rmf5AAAAAwEAAQ==');

    $google_key_bin = base64_decode(GOOGLE_KEY_B64);
    $modulus_len = unpack('Nl', $google_key_bin)['l'];
    $modulus_bin = substr($google_key_bin, 4, $modulus_len);
    $exponent_len = unpack('Nl', substr($google_key_bin, 4 + $modulus_len, 4))['l'];
    $exponent_bin = substr($google_key_bin, 4 + $modulus_len + 4, $exponent_len);
    $modulus = new phpseclib\Math\BigInteger($modulus_bin, 256);
    $exponent = new phpseclib\Math\BigInteger($exponent_bin, 256);

    $rsa = new phpseclib\Crypt\RSA();
    $rsa->loadKey(['n' => $modulus, 'e' => $exponent], phpseclib\Crypt\RSA::PUBLIC_FORMAT_RAW);
    $rsa->setEncryptionMode(phpseclib\Crypt\RSA::ENCRYPTION_OAEP);
    $rsa->setHash('sha1');
    $rsa->setMGFHash('sha1');
    $encrypted = $rsa->encrypt("{$email}\x00{$password}");

    $hash = substr(sha1($google_key_bin, true), 0, 4);
    return strtr(base64_encode("\x00{$hash}{$encrypted}"), '+/', '-_');
}

Answer 3

The user cannot directly access data in the hidden app folders, only the app can access them. This is designed for configuration or other hidden data that the user should not directly manipulate. (The user can choose to delete the data to free up the space used by it.)

The only way the user can get access to it is via some functionality exposed by the specific app.

Answer 4

public void retrieveContents(DriveFile file) {

    Task<DriveContents> openFileTask =
            getDriveResourceClient().openFile(file, DriveFile.MODE_READ_ONLY);



    openFileTask.continueWithTask(new Continuation<DriveContents, Task<Void>>() {
        @Override
        public Task<Void> then(@NonNull Task<DriveContents> task) throws Exception {
            DriveContents contents = task.getResult();

            try (BufferedReader reader = new BufferedReader(
                    new InputStreamReader(contents.getInputStream()))) {
                StringBuilder builder = new StringBuilder();
                String line;
                while ((line = reader.readLine()) != null) {
                    builder.append(line).append("\n");
                }

                Log.e("result ", builder.toString());
            }

            Task<Void> discardTask = MainActivity.this.getDriveResourceClient().discardContents(contents);
            // [END drive_android_discard_contents]
            return discardTask;
        }
    })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {

                }
            });


}
public void retrieveContents(DriveFile file) {

    Task<DriveContents> openFileTask =
            getDriveResourceClient().openFile(file, DriveFile.MODE_READ_ONLY);



    openFileTask.continueWithTask(new Continuation<DriveContents, Task<Void>>() {
        @Override
        public Task<Void> then(@NonNull Task<DriveContents> task) throws Exception {
            DriveContents contents = task.getResult();

            try (BufferedReader reader = new BufferedReader(
                    new InputStreamReader(contents.getInputStream()))) {
                StringBuilder builder = new StringBuilder();
                String line;
                while ((line = reader.readLine()) != null) {
                    builder.append(line).append("\n");
                }

                Log.e("result ", builder.toString());
            }

            Task<Void> discardTask = MainActivity.this.getDriveResourceClient().discardContents(contents);
            // [END drive_android_discard_contents]
            return discardTask;
        }
    })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {

                }
            });


}

Answer 5

to get all the file in app data try the code

private void listFiles() {
        Query query =
                new Query.Builder()
                        .addFilter(Filters.or(Filters.eq(SearchableField.MIME_TYPE, "text/html"),
                                Filters.eq(SearchableField.MIME_TYPE, "text/plain")))
                        .build();
        getDriveResourceClient()
                .query(query)

                .addOnSuccessListener(this,

                        new OnSuccessListener<MetadataBuffer>() {
                            @Override
                            public void onSuccess(MetadataBuffer metadataBuffer) {
                                //mResultsAdapter.append(metadataBuffer);

                                for (int i = 0; i <metadataBuffer.getCount() ; i++) {
                                    retrieveContents(metadataBuffer.get(i).getDriveId().asDriveFile());
                                }
                            }
                        }

                )
                .addOnFailureListener(this, new OnFailureListener() {
                    @Override
                    public void onFailure(@NonNull Exception e) {
                        Log.e(TAG, "Error retrieving files", e);
                        MainActivity.this.finish();
                    }
                });

    }

also you can download the content of file bye the following code

public void retrieveContents(DriveFile file) {

        Task<DriveContents> openFileTask =
                getDriveResourceClient().openFile(file, DriveFile.MODE_READ_ONLY);



        openFileTask.continueWithTask(new Continuation<DriveContents, Task<Void>>() {
            @Override
            public Task<Void> then(@NonNull Task<DriveContents> task) throws Exception {
                DriveContents contents = task.getResult();

                try (BufferedReader reader = new BufferedReader(
                        new InputStreamReader(contents.getInputStream()))) {
                    StringBuilder builder = new StringBuilder();
                    String line;
                    while ((line = reader.readLine()) != null) {
                        builder.append(line).append("\n");
                    }

                    Log.e("result ", builder.toString());
                }

                Task<Void> discardTask = MainActivity.this.getDriveResourceClient().discardContents(contents);
                // [END drive_android_discard_contents]
                return discardTask;
            }
        })
                .addOnFailureListener(new OnFailureListener() {
                    @Override
                    public void onFailure(@NonNull Exception e) {

                    }
                });


    }