Default SQLite DB encoding returns bad Cyrillic symbols

Question

Database table is created by simple:

public class DBHelper extends SQLiteOpenHelper {

    public static final String TABLE_ITEMS = "items";
    public static final String ITEMS_DB_CREATE = "create table "
                    + TABLE_ITEMS + "("
                    + ITEMS_ID + " integer primary key autoincrement, "
                    + ITEMS_NAME + " text, "
                    + ITEMS_DESCRIPTION + " text, "
                    + ITEMS_TYPE + " text);";

    public DBHelper(Context context) {
        super(context, DATABASE_NAME, null, DATABASE_VERSION);
    }

@Override
    public void onCreate(SQLiteDatabase database) {
        database.execSQL(ITEMS_DB_CREATE);
    }

Table is filled up by some data received from web API call to https://api.worldofwarships.ru/wows/encyclopedia/achievements/?application_id=demo

GSON is used for de-/serialization.

Code for fetching items:

public class ItemsFetcher {
    private SQLiteDatabase database;
    private DBHelper db_helper;

    public ItemsFetcher() {
    }

    public ItemsFetcher open() throws SQLException {
        db_helper = new DBHelper(App.getAppContext());
        database = db_helper.getWritableDatabase();
        return this;
    }

    public void close() {
        database.close();
    }

    String[] columns = {
            DBHelper.ITEMS_ID,
            DBHelper.ITEMS_NAME,
            DBHelper.ITEMS_DESCRIPTION
    };

    public Cursor fetchItems() {
        return database.query(DBHelper.TABLE_ITEMS, columns, null, null,
                null, null, DBHelper.ITEMS_ID);
    }

Then information is shown in several places around the app with:

@Override
public void bindView(View view, Context context, Cursor cursor) {
    if(locale.equals("ru") || locale.equals("uk") || locale.equals("be") || locale.equals("kk")) {
TextView tvTitle = (TextView) view.findViewById(R.id.tv_title);
tvTitle.setText(cursor.getString(cursor.getColumnIndexOrThrow(DBHelper.ITEMS_NAME)));
    }
    else {
TextView tvTitle = (TextView) view.findViewById(R.id.tv_title);
tvTitle.setText(cursor.getString(cursor.getColumnIndexOrThrow(DBHelper.ITEMS_NAME)));
   }
}

But some users experience the following instead of Cyrillic text:

What could the problem be related to? I've never used any special encoding stuff. According to quite authoritative answer here https://stackoverflow.com/a/4276758/1214162, DB should be set to UTF by default.

Important finding

It happens during json deserialization for the 2+ time.

If the app is installed from scratch and it loads information for the first time - everything is okay with fields containing Cyrillic.

If then it finds that a new data is available on the server, it calls drop + re-create for tables and starts downloading new jsons.

Here the ÐÑÑÐ¼Ð°Ð½ÑÐ appears for the same Cyrillic fields!

@Rod_Algonquin actually, here is the real API url used, content from which results in such a bad text: https://api.worldofwarships.ru/wows/encyclopedia/achievements/?application_id=demo — Ivan Fazaniuk, Dec 16 '15 at 00:24
There is an encoding bug somewhere in your code. Which you have not shown. — CL., Dec 16 '15 at 08:05
@CL. Added more code to show where the potential place for the bug is. — Ivan Fazaniuk, Dec 16 '15 at 18:10
There's nothing wrong with your code. I suspect the web API data. — CL., Dec 16 '15 at 20:07
I've just discovered it happens during json deserialization for the 2+ time. If the app is installed from cratch and loads data for the first time - everythins is ok. If then it finds that a new data is available on the server, it calls drop and re-create for tables starts downloading json. And here the ÐÑÑÐ¼Ð°Ð½ÑÐ appears! — Ivan Fazaniuk, Dec 17 '15 at 02:18

score 0 · Accepted Answer · answered Dec 18 '15 at 18:18

0

API was returning 304 NOT MODIFIED with null in Charset.

Resolved with changing

String json = new String(response.data, HttpHeaderParser.parseCharset(response.headers));

to

String json = new String(response.data, "UTF-8");

answered Dec 18 '15 at 18:18

Ivan Fazaniuk

1,062
3
11
26

Default SQLite DB encoding returns bad Cyrillic symbols

1 Answers1