2

Here's our situation:

We have some native code that scans user-defined directories at certain times. The goal is to collect data about specific types of file and build an SQLite database from it. There could be tens of thousands of files to query.

The problem is that each time a file is found, the data collected about that file must be ushered across the JNI boundary to Java-land before being comitted to the database via the SQLite classes provided by the Android SDK. This takes far too long.

The proposed solution is to download the SQLite source code, link it into our project and build the database natively as described here: SQLite with Android NDK. This would eliminate the time taken to pass the data from the C++ layer to the Java layer entirely.

The question is: if this is done, can the Android SDK (Java layer) still be used to open the database, obtain Cursor objects etc. as normal? We can keep the database extremely simple; no foreign keys, triggers, constraints etc. Although indexes are very desirable.

We have considered pipes and sockets to move the data across the JNI boundary. However, named pipes are not supported on Fat32 file systems (so this is not useable incase the user moves the application to a Fat32 formatted SD card). Sockets are also not usable because we will have to include internet permissions in the manifest which we feel would look suspicious.

If anybody has any information on this, we'd love to hear from you.

Many thanks, P.

Community
  • 1
  • 1
protectedmember
  • 349
  • 5
  • 20

1 Answers1

4

Just to follow up, we took the plunge and threw together a test application that compiled the latest SQLite Amalgamation (3.7.13) source code. We have found that we are indeed able to interface with the natively created database via the Android Java SDK.

Performance wise, this was a great leap! By inserting our records at the native layer we found a massive performance increase. Previously, inserting 20,000 records into a non-complex database took approximately 4 minutes. Some simple profiling showed that pretty much the entire 4 minutes was spent passing the data from the native layer to the Java layer.

Now, the insertions take between 4 and 6 seconds. We are also able to open, query etc. the database with the Java API as per normal.

Performance tweaks to note are transactions and moving the journal to memory using:

pragma journal_mode=memory;

We decided not to turn off the synchronisation for stability reasons, plus we were more than happy with the performance we had already achieved.

Hopefully, others will find this advice useful.

P

protectedmember
  • 349
  • 5
  • 20
  • How did you handled the DB lock? If Im accessing DB from few threads in Android and also in native, will there be a chance of DB corruption? Did you face any such issues? I know that SQL is thread safe, but im not sure if can really be thread safe across Java and C++. I know this is very old post, but I would really appreciate if you can provide some thoughts here – Santosh Nov 26 '14 at 14:44
  • I'm sorry, but threading wasn't an issue for us since we were inserting all our records at the native layer before any other part of the application (Java layer) began to read/write/delete/modify those records. – protectedmember Aug 19 '15 at 08:00
  • @protectedmember How have you managed to profile this? - Some simple profiling showed that pretty much the entire 4 minutes was spent passing the data from the native layer to the Java layer. – b-boy sh1ft Sep 28 '17 at 13:49
  • @b-boysh1ft I@m sorry but this was many years ago and I@m no longer working for the same company. I can't remember how we profiled, not do I have access to any code to check for you. – protectedmember Oct 03 '17 at 10:54