1

Tentatively, my app will be retrieving at least 86400 rows of data everyday, as we have a total of 10 channels, with each channel producing 8640 rows of data (1 row is inserted every 10 seconds throughout the day).

(Done on mobile at the moment due to some major limitations. Will be migrating these intense processing to a middleware somewhere in the future)

The problem now is halfway through the retrieval, my app will crash due to the 1MB size limit of the cursor. The following is the method that is used to do the retrieval.

public Cursor getUnconsolidatedReadings(String channelID) {
        String[] channelIDs = {channelID};
        db = this.getReadableDatabase();
        String sqlRetrieve = "SELECT ChannelID, TimeStamp, ID, RMSCurrent, RealPower FROM Readings WHERE Consolidated = 0 AND ChannelID = ?";
        Cursor cursor = db.rawQuery(sqlRetrieve, channelIDs);

        return cursor;
    }

Is there any better way to do such large data retrieval?

Zhi Kai
  • 1,549
  • 1
  • 13
  • 31
  • do you have to store such huge BLOBs in your sqlite database? – pskink Aug 11 '16 at 05:36
  • Unfortunately for now, yes. We had no choice but to store and conduct major processing on the phone itself. – Zhi Kai Aug 11 '16 at 05:40
  • 2
    This may help you http://stackoverflow.com/questions/9873869/android-how-to-query-huge-database-in-android-cursor-size-is-limited-to-1mb – Apurva Aug 11 '16 at 05:40
  • i mean use `nosql` like [this](https://github.com/nhachicha/SnappyDB) for example, of course you can combine it with sqlite and store BLOBs only in SnappyDB – pskink Aug 11 '16 at 05:41
  • Thanks both, I will give your suggestions a try to see the performance. – Zhi Kai Aug 11 '16 at 05:48

0 Answers0