Missing records due to skipped frames

Question

The app im working with is getting data from a .csv (20k-30k records) from a server and it needs to persist the data into an SQLiteDatabase.

It works but some records are missing and appeared that it have been skipped.

I/Choreographer( 2555): Skipped 46 frames!  The application may be doing too much work on its main thread.

I know that this error says that the memory consumption is very high due to heavy load. Is there a more efficient way in persisting data in SQLiteDatabase rather than the classic accessing of CSV and processing it from there?

Code for writing in DB

String sql = "INSERT INTO " + tableName
                + " VALUES (?,?,?,?,?,?);";
        SQLiteDatabase db = openHelper.getWritableDatabase();
        SQLiteStatement statement = db.compileStatement(sql);

        try {
            db.beginTransaction();
            String[] sa = null;
            for (final String csvline : arrCSV) {
                statement.clearBindings();
                sa = csvline.split(",");
                if(sa.length==6){
                    statement.bindString(1, sa[0]);
                    statement.bindString(2, sa[1]);
                    statement.bindString(3, sa[2]);
                    statement.bindString(4, sa[3]);
                    statement.bindString(5, sa[4]);
                    statement.bindString(6, sa[5]);
                }
                statement.execute();
            }
            db.setTransactionSuccessful();
            Log.d("Transaction", "Successful");
        }catch(Exception e){
            e.printStackTrace();
        }finally {
            statement.releaseReference();
            statement.close();
            db.endTransaction();            
            db.releaseMemory();
        }

UPDATE

The missing records were not loaded in the Collection. Is the skipping of frames the culprit here? The loading in the collection is just a simple parsing of a csv file and non replicable at times so Im assuming it is due to the skipping of frames.

Answer 1

I believe the issue is not linked to skipping frames and < 100 frames is considered a small/insignificant number. At least according to The application may be doing too much work on its main thread.

I frequently see it and has never been the cause of any issues. I've even seen it basically doing nothing other than returning a result from an activity to an activity that just starts the second activity.

As you have commented, the number of elements that result from the split is on occasion not 6. The issue is likely that the insert is not happening on such an occasion, perhaps due to constraints (without seeing how the columns are defined only guesses could be made).

However, you appear to consider that each line in csvline should be split into 6 elements. You should thus investigate as to why not?

To investigate I'd suggest getting details of the original data before the split and the resultant data after the split whenever the number of elements created by the split is not 6. e.g. by changing :-

            sa = csvline.split(",");
            if(sa.length==6){
                statement.bindString(1, sa[0]);
                statement.bindString(2, sa[1]);
                statement.bindString(3, sa[2]);
                statement.bindString(4, sa[3]);
                statement.bindString(5, sa[4]);
                statement.bindString(6, sa[5]);
            }
            statement.execute();

to

            sa = csvline.split(",");
            if(sa.length==6){
                statement.bindString(1, sa[0]);
                statement.bindString(2, sa[1]);
                statement.bindString(3, sa[2]);
                statement.bindString(4, sa[3]);
                statement.bindString(5, sa[4]);
                statement.bindString(6, sa[5]);
            } else {
                Log.d("NOT6SPLIT","CSVLINE WAS ===>" + csvline + "<===");
                Log.d("NOT6SPLIT","CSVLINE WAS SPLIT INTO " + Integer.toString(sa.length) + " ELEMENTS :-");
                for(String s: sa) {
                    Log.d("NOT6SPLIT","\tElement Value ===>" + s + "<===");
                }
            }
            statement.execute();

Changing statement.execute() to :-

           if (statement.executeInsert() < 1) {
               Log.d("INSERTFAIL","Couldn't insert where CSVLINE was ===>" + csvline + "<===");
           }

May also assist ('executeInsert' returns the rowid of the inserted record, else -1, not sure of the consequences of a table defined with WITHOUT ROWID).

It wouldn't surprise me at all if the issue boils down to your data containing characters that split considers special or metacharaceters:-

there are 12 characters with special meanings:

• the backslash \,
• the caret ^,
• the dollar sign $,
• the period or dot .,
• the vertical bar or pipe symbol |,
• the question mark ?,
• the asterisk or star *,
• the plus sign +,
• the opening parenthesis (,
• the closing parenthesis ),
• the opening square bracket [,
• and the opening curly brace {,

These special characters are often called "metacharacters". Most of them are errors when used alone.