0

I am developing an Android app with offline search functionality. This requires me to read in a dictionary file that has approximately 170,000 entries.

I am facing severe performance issues and initially thought it was due to my code having String.match(regex) looping through the ArrayList which I had read the data into.

However, digging deeper, I found that the main issue was actually data I/O. It took ~10,000 ms just to read the dictionary file in via BufferedReader + InputStream, without performing any searches. I've tried other means of reading data in such as using a StringBuilder but they don't seem to help much.

What are some possible solutions to this problem?

Some points to consider:

  • The searches are recursive
  • The app has to stay offline
  • The entries are basically lines of text, albeit in an East Asian language (increasing the difficulty of the search, since entire sentences can be a single string.)

The standard code I was using for reading in the data was:

InputStream is = getResources().openRawResource(R.raw.data);     
BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
     while(br.readLine() != null){
         blahblah....
        }
Reuben L.
  • 2,806
  • 2
  • 29
  • 45
  • 2
    Why are you not using a database? – jlordo Jun 02 '13 at 15:59
  • to be honest, I've not really tried reading from an existing database, usually its creating the DBs at runtime. perhaps I should do that. – Reuben L. Jun 02 '13 at 16:10
  • you create 170.000 records at runtime every time? Are they the same, or random? – jlordo Jun 02 '13 at 16:11
  • i meant that previous apps i made usually only created databases and not retrieved data from existing databases. in this particular case, there is no need for writing to the DB, just retrieval only. – Reuben L. Jun 02 '13 at 16:13
  • http://stackoverflow.com/questions/513084/how-to-ship-an-android-application-with-a-database seems like its possible but not straightforward – Reuben L. Jun 02 '13 at 17:22

1 Answers1

0

I agree with Reuben L. I would use sqlite database instead of parsing text file.

In one of my app I put all villages and cities in Czech Republic (over 200000 records) into database for fast offline search. It could find list of cities based on name prefix in 1 sec max.

To have it even faster I introduced db indexes. I added one column and filled there the first letter of city name. Then I could search the cities by prefix like this:

select * from cities where firstLetter = ? and cityName like ?;

It improved performance more than twice.

smuk
  • 321
  • 2
  • 12