How to design a file system based on a singlefile on Android and iOS?

Question

Recently, I am working on a project which will have thousands of small files to manage. The whole system will randomly read/write those files. And parts of them are downloaded from networks at the moment that it is needed. For performance and system limitations, I could only keep 10~20 file descriptors/handles. So I decided to create a large single file as my 'disk'. And then dynamic read and write files into it.

Here is my design:

1. A file header for serialization of meta-info
2. An index to save offset of every little file
3. A bunch of 'pre-allocate segments' to save files
4. Support multi-thread accessing

Finally, It looks like a virtual file system. But I'm not sure is it safe to read/write the same file at the same time on Android and iOS. And I'm totally no idea how to manage the fragment of my 'pre-allocate segments'.

Is there any best practice or frameworks that I can reference? Am I on the right way?

Answer 1

First off:

I'm not sure is it safe to read/write the same file at the same time on Android and iOS

One the one hand, it is safe, in the sense that it won't break the OS, the filesystem, other files, or other applications not using this file. Your concurrent reads and writes will also not fail.

On the other hand, concurrent reads and writes will not be atomic, so if you overwrite a 10K file in one thread while another thread is reading it, you might get half-new-half-old data.

Secondly, this problem of non-atomicity will also apply to your single-file filesystem framework. In any case, this can be mitigated (by you or the the framework) by locking the byte ranges being read & written.

Filesystems do support byte range locking on files. I could easily find the Android docs. iOS seems to take a different approach.

iOS seems to be taking an event-based approach to changes to parts of files. But if you just use the filesystem directly rather than building your own filesystem in a single file, all you need to do is to atomically replace files.
This approach also has an equivalent in android.

Answer 2

In the Java (Android world) you can look at :

• RandomAccessFile : good old way of reading and writing at any position of a given file
• FileChannel : thread safe method for doing it (but you need to manage yourself multiple concurrent write to the same file by managing multiple FileChannel instances over the same file see Concurrency of RandomAccessFile in Java)

What you're looking for is close to what most of the bitorrent clients does (at least on good old hard drive to avoid fragmentation), creating a file of a given size (reserving the space for the whole download on disk) and then downloading and serving segments of it over the network !

According to your design, segments are just a matter of how many bytes you reserve in your file for managing allocations. You can read about file systems on the web !

don't know much regarding the iOs world...

Answer 3

I am not an expert on Android/Ios but I think that in your current case the problem will be to use multiple threads on a io operation. My advice if you want to follow down this place would be to add an intermediate memory buffer that could handle the update/deletion and asynchronously write on disk. Another advice would be to try to use an sqlite db (a db with a single file) that is already doing all the heavy work for you.

Answer 4

Put all files in a database. Sqlite for Android. Then they are in one file.