How to get access to all SD-cards, using the new Lollipop API?

Question

background

Starting with Lollipop, apps can get access to real SD-cards (after it was not accessible on Kitkat, and was not officially supported yet worked on previous versions), as I've asked about here.

The problem

Because it has become quite rare to see a Lollipop device that supports SD-card and because the emulator doesn't really has the ability (or does it?) to emulate an SD-card support, it took me quite a while to test it.

Anyway, it seems that instead of using the normal File classes to access the SD-card (once you got a permission for it), you need to use Uris for it, using DocumentFile .

This limits the access to the normal paths, as I can't find a way to convert the Uris to paths and vice versa (plus it's quite annoying). It also means that I don't know how to check if the current SD-card/s are accessible, so I don't know when to ask the user for permission to read/write to it (or to them).

What I've tried

Currently, this is how I get the paths to all SD-cards:

  /**
   * returns a list of all available sd cards paths, or null if not found.
   *
   * @param includePrimaryExternalStorage set to true if you wish to also include the path of the primary external storage
   */
  @TargetApi(Build.VERSION_CODES.HONEYCOMB)
  public static List<String> getExternalStoragePaths(final Context context,final boolean includePrimaryExternalStorage)
    {
    final File primaryExternalStorageDirectory=Environment.getExternalStorageDirectory();
    final List<String> result=new ArrayList<>();
    final File[] externalCacheDirs=ContextCompat.getExternalCacheDirs(context);
    if(externalCacheDirs==null||externalCacheDirs.length==0)
      return result;
    if(externalCacheDirs.length==1)
      {
      if(externalCacheDirs[0]==null)
        return result;
      final String storageState=EnvironmentCompat.getStorageState(externalCacheDirs[0]);
      if(!Environment.MEDIA_MOUNTED.equals(storageState))
        return result;
      if(!includePrimaryExternalStorage&&VERSION.SDK_INT>=VERSION_CODES.HONEYCOMB&&Environment.isExternalStorageEmulated())
        return result;
      }
    if(includePrimaryExternalStorage||externalCacheDirs.length==1)
      {
      if(primaryExternalStorageDirectory!=null)
        result.add(primaryExternalStorageDirectory.getAbsolutePath());
      else
        result.add(getRootOfInnerSdCardFolder(externalCacheDirs[0]));
      }
    for(int i=1;i<externalCacheDirs.length;++i)
      {
      final File file=externalCacheDirs[i];
      if(file==null)
        continue;
      final String storageState=EnvironmentCompat.getStorageState(file);
      if(Environment.MEDIA_MOUNTED.equals(storageState))
        result.add(getRootOfInnerSdCardFolder(externalCacheDirs[i]));
      }
    return result;
    }


private static String getRootOfInnerSdCardFolder(File file)
  {
  if(file==null)
    return null;
  final long totalSpace=file.getTotalSpace();
  while(true)
    {
    final File parentFile=file.getParentFile();
    if(parentFile==null||parentFile.getTotalSpace()!=totalSpace)
      return file.getAbsolutePath();
    file=parentFile;
    }
  }

This is how I check what Uris I can reach:

final List<UriPermission> persistedUriPermissions=getContentResolver().getPersistedUriPermissions();

This is how to get access to the SD cards:

startActivityForResult(new Intent(Intent.ACTION_OPEN_DOCUMENT_TREE),42);

public void onActivityResult(int requestCode,int resultCode,Intent resultData)
  {
  if(resultCode!=RESULT_OK)
    return;
  Uri treeUri=resultData.getData();
  DocumentFile pickedDir=DocumentFile.fromTreeUri(this,treeUri);
  grantUriPermission(getPackageName(),treeUri,Intent.FLAG_GRANT_READ_URI_PERMISSION|Intent.FLAG_GRANT_WRITE_URI_PERMISSION);
  getContentResolver().takePersistableUriPermission(treeUri,Intent.FLAG_GRANT_READ_URI_PERMISSION|Intent.FLAG_GRANT_WRITE_URI_PERMISSION);
  }

The questions

1. Is it possible to check if the current SD-cards are accessible, which aren't , and somehow ask the user to get permission to them?

2. Is there an official way to convert between the DocumentFile uris and real paths? I've found this answer, but it crashed in my case, plus it looks hack-y.

3. Is it possible to request permission from the user about a specific path? Maybe even show just the dialog of "do you accept yes/no ? " ?

4. Is it possible to use the normal File API instead of the DocumentFile API once the permission was granted?

5. Given a file/filepath, is it possible to just request a permission to access it (and check if it's given before), or its root path ?

6. Is it possible to make the emulator have an SD-card? Currently, it has "SD-card" being mentioned, but it works as the primary external storage, and I'd like to test it using the secondary external storage, in order to try and use the new API.

I think that for some of those questions, one helps a lot to answer the other.

Answer 1

Is it possible to check if the current SD-cards are accessible, which aren't , and somehow ask the user to get permission to them?

There are 3 parts to this: detecting, what cards there are, checking if a card is mounted, and asking for access.

If device manufacturers are good guys, you can get list of "external" storage by calling getExternalFiles with null argument (see the associated javadoc).

They may not be good guys. And not everyone have newest, hottest, bug-free OS version with regular updates. So there may be some directories, that aren't listed there (such as OTG USB storage etc.) Whenever in doubt, you can get full list of OS mounts from file /proc/self/mounts. This file is part of Linux kernel, it's format is documented here. You can use contents of /proc/self/mounts as backup: parse it, find usable filesystems (fat, ext3, ext4 etc.) and remove duplicates with output of getExternalFilesDirs. Offer remaining options to users under some non-obtrusive name, something like "misc directories". That will give you every possible external, internal, whatever storage ever.

---

EDIT: after giving the advice above I have finally tried to follow it myself. It worked fine so far, but beware that instead of /proc/self/mounts you are better off parsing /pros/self/mountinfo (the former is still available in modern Linux, but later is a better, more robust replacement). Also make sure to account for atomicity issues when making assumptions based on contents of the mounts list.

---

You can naively check, if the directory is readable/writable by calling canRead and canWrite on it. If this succeeds, no point in doing extra work. If it does not, you either have a persisted Uri permission or don't.

"Getting permission" is an ugly part. AFAIK, there is no way to do that cleanely within SAF infrastructure. Intent.ACTION_PICK sounds like something that may work (since it accepts an Uri, from which to pick), but it does not. Maybe, this can be considered a bug and should be reported to Android bug tracker as such.

Given a file/filepath, is it possible to just request a permission to access it (and check if it's given before), or its root path ?

This is what ACTION_PICK is for. Again, the SAF picker does not support ACTION_PICK out of box. Third-party file managers may, but very few of them will actually grant you real access. You may report this as bug too, if you feel like it.

---

EDIT: this answer was written before Android Nougat came out. As of API 24, it is still impossible to request fine-grained access to the specific directory, but at least you can dynamically request access to the entire volume: determine the volume, containing the file, and request access using either getAccessIntent with null argument (for secondary volumes) or by requesting WRITE_EXTERNAL_STORAGE permission (for the primary volume).

---

Is there an official way to convert between the DocumentFile uris and real paths? I've found this answer, but it crashed in my case, plus it looks hack-y.

No. Never. You shalt forever remain subservient to whims of Storage Access Framework creators! evil laughter

Actually, there is a much simpler way: just open the Uri and check filesystem location of created descriptor (for simplicity, Lollipop-only version):

public String getFilesystemPath(Context context, Uri uri) {
  ContentResolver res = context.getContentResolver();

  String resolved;
  try (ParcelFileDescriptor fd = res.openFileDescriptor(someSafUri, "r")) {
    final File procfsFdFile = new File("/proc/self/fd/" + fd.getFd());

    resolved = Os.readlink(procfsFdFile.getAbsolutePath());

    if (TextUtils.isEmpty(resolved)
          || resolved.charAt(0) != '/'
          || resolved.startsWith("/proc/")
          || resolved.startsWith("/fd/"))
    return null;
  } catch (Exception errnoe) {
    return null;
  }
}

If the method above returns a location, you still need access to use it with File. If it does not return a location, the Uri in question does not refer to file (even temporary one). It may be a network stream, Unix pipe, whatever. You can get version of method above for older Android versions from this answer. It works with any Uri, any ContentProvider—not just SAF—as long as the Uri can be opened with openFileDescriptor (e.g. it is from Intent.CATEGORY_OPENABLE).

Note, that approach above can be considered official, if you consider any part of official Linux API as such. A lot of Linux software uses it, and I have also seen it to be used by some AOSP code (such as in Launcher3 tests).

---

EDIT: Android Nougat introduced number of security changes, most notably changes to permissions of application private directories. This means, that since API 24 the snippet above will always fail with Exception when the Uri refers to the file inside an application private directory. This is as intended: you are no longer expected to know that path at all. Even if you somehow determine the filesystem path by other means, you can not access the files using that path. Even if the other application cooperates with you and changes the permission of file to world-readable, you still won't be able to access it. This is because Linux does not allow access to files, if you don't have search access to one of directories in path. Receiving a file descriptor from ContentProvider thus is the only way to access them.

---

Is it possible to use the normal File API instead of the DocumentFile API once the permission was granted?

You can not. Not on Nougat at least. Normal file API goes to Linux kernel for permissions. According to Linux kernel, your external SD card has restrictive permissions, that prevent your app from using it. The permissions, given by Storage Access Framework are managed by SAF (IIRC stored in some xml files) and kernel knows nothing about them. You have to use intermediate party (Storage Access Framework) to gain access to external storage. Note, that Linux kernel has it's own mechanism for managing access to directory subtrees (it is called bind-mounts), but Storage Access Framework creators either don't know about it or don't want to use it.

You can get some degree of access (pretty much anything, that can be done with File) by using file descriptor, created from Uri. I recommend you to read this answer, it may have some helpful information about using file desriptors in general and in relation to Android.

Answer 2

This limits the access to the normal paths, as I can't find a way to convert the Uris to paths and vice versa (plus it's quite annoying). It also means that I don't know how to check if the current SD-card/s are accessible, so I don't know when to ask the user for permission to read/write to it (or to them).

Starting from API 19 (KitKat) nonpublic Android class StorageVolume may be used to get some answers on your question by means of a reflection:

public static Map<String, String> getSecondaryMountedVolumesMap(Context context) throws NoSuchMethodException, InvocationTargetException, IllegalAccessException {
    Object[] volumes;

    StorageManager sm = (StorageManager) context.getSystemService(Context.STORAGE_SERVICE);
    Method getVolumeListMethod = sm.getClass().getMethod("getVolumeList");
    volumes = (Object[])getVolumeListMethod.invoke(sm);

    Map<String, String> volumesMap = new HashMap<>();
    for (Object volume : volumes) {
        Method getStateMethod = volume.getClass().getMethod("getState");
        String mState = (String) getStateMethod.invoke(volume);

        Method isPrimaryMethod = volume.getClass().getMethod("isPrimary");
        boolean mPrimary = (Boolean) isPrimaryMethod.invoke(volume);

        if (!mPrimary && mState.equals("mounted")) {
            Method getPathMethod = volume.getClass().getMethod("getPath");
            String mPath = (String) getPathMethod.invoke(volume);

            Method getUuidMethod = volume.getClass().getMethod("getUuid");
            String mUuid = (String) getUuidMethod.invoke(volume);

            if (mUuid != null && mPath != null)
                volumesMap.put(mUuid, mPath);
        }
    }
    return volumesMap;
}

This method gives you all mounted non-primary storages (including USB OTG) and maps their UUID to their path that will helps you to convert DocumentUri to real path (and vise versa):

@TargetApi(Build.VERSION_CODES.LOLLIPOP)
public static String convertToPath(Uri treeUri, Context context) throws NoSuchMethodException, InvocationTargetException, IllegalAccessException {
    String documentId = DocumentsContract.getTreeDocumentId(treeUri);
    if (documentId != null){
        String[] split = documentId.split(":");
        String uuid = null;
        if (split.length > 0)
            uuid = split[0];
        String pathToVolume = null;
        Map<String, String> volumesMap = getSecondaryMountedVolumesMap(context);
        if (volumesMap != null && uuid != null)
            pathToVolume = volumesMap.get(uuid);
        if (pathToVolume != null) {
            String pathInsideOfVolume = split.length == 2 ? IFile.SEPARATOR + split[1] : "";
            return pathToVolume + pathInsideOfVolume;
        }
    }
    return null;
}

It seems Google does not going to give us better way to handle with their ugly SAF.

EDIT: Seems this approach is useless in Android 6.0...